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were asked to complete a search on the ERIC CD-ROM; and data were 
collected via observations, a think'-a.oud protocol, and a 
questionnaire. Data were analyzed to describe and categorize the 
subjects* experience with the user interface. The standard Macintosh 
interface as implemented did not win over any users who began with a 
preference for IBM personal computers or compatibles. Experienced 
Macintosh users gave the product high marks, but users at all 
experience levels had problems with the interaction provided. Novice 
users expressed a need for more direction and clarity, and 
experienced users looked for a more direct way to conduct searches. 
The most apparent negative attitudes concerned the error messages 
provided. Four appendixes are included. Appendix A contains the coded 
tape transcripts. Appendix B is the interview script. The site 
software license agreement for the computerized Questionnaire for 
User Interface Satisfaction is presented in Appendix C, and a 
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This study examines the experience of novice users when 
searching SilverPlatter's ERIC CD-ROM on the Macintosh. Ten 
students from an introductory master's level course in library and 
information science were recruited as volunteer subjects. Subjects 
were asked to complete a search on the ERIC CD-ROM, and data was 
collected via observation, think-aloud protocol, and questionnaire. 
Data was analyzed to describe and categorize the subjects' 
experiences with the user interface. 
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Introduction 
Problem Statement 

The use of compact disc, read-only-memory (CD-ROM) 
technology in the 1980s has led to a number of improvements in 
information retrieval. CD-ROMs are a space saving technology that, 
with search software, allow fast access to information with a variety 
of search options (Zink, 1991, p. 16). "Publishers of hard copy didn't 
worry about how the eventual user would use their product, but for 
CD-ROM publishers this realization created the concept of user 
interface" (Helgerson, 1986, p. 71). Shaw (1991) defines the human- 
computer interface as "what the user sees, hears, and touches in 
interacting with a computer system" (p. 155). It is this interface that 
may hinder, frustrate, or prevent the user from achieving a 
successful search. 

Nicholls, Han, Stafford, and Whitridge (1990) consider a CD- 
ROM product to have three parts: database, retrieval engine, and 
user interface. They combine the retrieval engine and user interface 
to define the access software, that software is "largely characterized 
by variety and rapid change" (pp. 41, 44). Most CD-ROM interfaces 
were originally designed "to emulate the functions and features of 
the major online access software," (Herther, 1986, p. 94) but now 
that hardware has begun to stabilize, the user interface is one of the 
places that a publisher can distinguish its product in the 
marketplace. Figure 1 shows the proliferation of distinct retrieval 
software on CD-ROM products from 1987 to 1989 (Nicholls & Elshout, 
1990). "Most of these (67%) are proprietary programs used on a 
single product only; 79% of all software is used on only one or two 
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products" (p. 22). CD-ROM technology presents itself as a rich 
resource for the presentation of text, audio, and graphics, and with 
the diversity of interfaces that have been spawned, it is time for the 
industry to turn to quality (Nicholls & Elshout, 1990, p. 22; Shaw, 
1991, p. 168). 

Figure 1 

Access Software 
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Although user satisfaction for CD-ROMs is typically high, 
studies have shown that training and experience are still required to 
achieve accurate use of the products. Yet, users feel that they need 
neither training nor instructions for proper CD-ROM use 
(Puttapithakporn, 1990, p. 196). An interface designed to address 
problems of a novice searcher will increase user search success and 
provide useful information for future product design. 

The user interface involves the hardware as well as the 
software. This combination makes the Apple Macintosh (Mac) 
an interesting platform to study when used with CD-ROM. 
Traditionally, the IBM personal computer (PC) relies on a 
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command line interface and keyboard, while the Mac has 
defined standards using windows, pull-down menus and a 
mouse. In designing access software for CD-ROM to run on the 
Mac, SilverPiatter Vice President Christopher Pooley noted that 
it was hard to develop and support both platforms for one 
product, and that there was a question of whether or not the 
interface for the product should, in fact, be changed from that 
on the PC. Pooley states that "there are areas where we feel 
that the Mac standards can't be violated and others where we 
feel that there is some latitude in terms of integrating features 
from the users' perspective" (Pemberton, 1989, pp. 20-21). 

A new interface was, in fact, created and named 
MacSPIRS. Although it assumes familiarity with the Mac, its 
standards, and terminology, the user manual offers the 
following advice: "we think that MacSPIRS is very easy to use, 
but we are also aware that people who are not already familiar 
with information retrieval might need a little assistance to help 
them start out" (SilverPiatter, 1990, p. 2). The manual presents 
the basic techniques for information retrieval, functions of the 
access software, and an explanation of how to access the help 
features. The documentation of the product, though an 
important point to be evaluated, is beyond the scope of this 
study. The "very easy to use" claim, however is one that will 
be studied, described, and quantified here. 

Zink (1991) asks for more critical evaluations of CD-ROM 
interfaces in the literature to aid in both library collection 
development and industry awareness of the importance of the 
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interface. While the data that exists on CD-ROM is usually available 
in print, it is the powerful search capabilities provided by the access 
software that provide the major benefit. "The major challenge facing 
software authors is to make the J/Oftware designer a skilled 
communicator, not to make the average person computer literate. As 
any writer knows, the primary task of any communication falls on 
the originator" (p. 17). Shneiderman (1987) challenges designers to 
"go beyond the vague notion of 'user friendliness' and probe deeper 
than a checklist of subjective guidelines. They must have a thorough 
understanding of the diverse community of users and the tasks that 
must be accomplished. Moreover, they muit have a deep 
commitment to serving the users" (p. 9). 

It is this combination of interface and user which lends 
itself to what Mellon (1990) defines as naturalistic inquiry. 
She states the case as follows: 

Naturalistic inquiry is particularly relevant to 
librarianship. In this age of automation, it is necessary 
for librarians to keep reminding themselves that theirs is 
a profession aimed at bringing information to people. It 
is not enough to develop better methods of storing and 
retrieving the information: we must remember that the 
purpose of storage and retrieval is to satisfy the needs of 
people. And the focus of naturalistic inquiry is people- 
how and why they think and act as they do. (p.3) 

This qualitative study will examine the experience of novice 

users when using the MacSPIRS interface to search ERIC. Data will be 

gathered by observation, think-aloud protocol, and questionnaire. 

Data will be analyzed to describe and categorize the subjects' 

experiences with the user interface. 



Literature Review 

"The actual dawn of user interface design first happened when 
computer designers finally noticed, not just that end users had 
functioning minds, but that a better understanding of how those 
minds worked would completely shift the paradigm of interaction" 
(Kay, 1990, p. 192). In the 1970's the study of human interface 
began as an art and a science with the development of new types of 
computer input and output devices (Rheingold, 1985, p. 249). Now 
interface design is on the minds of computer scientists, psychologists, 
software designers, and many professionals, including librarians. 
Shaw (1991) found the following terms in her review of this 
emerging field: "cognitive engineering, cognitive ergonomics, 
computer-human interaction/interface (CHI), convivial computing, 
cooperative interface, human-computer interaction/interface (HCI), 
person-machine interface, software ergonomics, usability 
engineering, user friendly/cordial/oriented/centered, and user 
interface" (pp. 178-9). 

The diversity of terms is rivaled by the number of interfaces 
and theories that exist. Kay (1990) urges designers not to transfer 
the way we use paper to the software interface, but to create "user 
illusion" (p. 199). He has involved children from the beginning of his 
research because he insisted that the interface had to be as easy to 
understand as it was powerful (Rheingold, 1985, p. 249). Kay (1990) 
drew from Piaget's theory of children's mental development these 
three stages: enactive, iconic, and symbolic. Good design should 
integrate these philosophies-no one alone is "right" (pp. 194-5). The 
enactive stage is embodied by the use of the mouse, being able to 



manipulate by moving and pointing. The iconic stage is represented 
by the use of images, that is, icons and windows, so that the user can 
easily recognize functions or ideas. Finally, the symbolic stage is 
accomplished through the use of object-oriented programming to tie 
together long chains of reasoning (p. 196). These principles became 
the standards for the Mac interface which would be carried through 
to many applications. Kay, however, acknowledged that it would not 
always be done well. He compared ''applying Macintosh style to 
poorly designed applications and machines" to "putting Bearnaise 
sauce on a hotdog" (p. 191). 

It is the combination of the hardware and software aspects of 
the interface that make the Mac so interesting to evaluate with 
respect to the effect on novice searchers of CD-ROMs. The Mac, since 
its inception, has incorporated a graphical user interface (GUI) and a 
mouse which are now available on IBM PCs and compatibles via the 
software product Microsoft Windows. MacSPIRS makes use of the 
GUI but also draws from other interface styles. 

Shaw ( 1991) describes the GUI as follows: 

Some writers have used the playful and slightly 
pejorative term WIMP to refer to interfaces that use 
windows, icons, menus/mice, and pointers/pull-down 
menus. A less partisan term is 'graphic user interfaces' 
(GUIs). Graphic interfaces that incorporate icons are 
standard for many microcomputer applications, allowing 
the user to select a picture of the desired action rather 
than having to remember specific commands, (p. 160) 

This selection is accomplished via the mouse, an indirect control 

away from the screen surface. It "is attractive because the users can 

avoid learning commands, reduce the chance of typographic errors 



on a keyboard, and keep their attention on the display" 
(Shneiderman, 1987, p. 237). However, Ziegler and Fahnrich 
reviewed empirical investigations and found consistent problems for 
the novice user with mouse manipulation (Shaw, 1991, p. 164). 
Nonetheless, Percival and Morrow (1989) assert that "increasingly, 
the ne v computer literate generation is being weaned on graphical 
user interfaces (GUIs) and is coming to expect and demand that kind 
of intuitive ease of use" (p. 266). 

Another type of interface employed in MacSPIRS is Menu- 
Based Natural Language (MBNL). Here, words and phrases are 
displayed so the user can point to them with the mouse to select 
them. No typing is necessary, reducing chances for typographical 
errors, and no special syntax is required (Hendrickson, 1989, p. 217). 

"It is generally accepted that a good interface is internally 
consistent and that consistency among interfaces provides for greater 
transfer of skills to new applications. Poison cites the Apple 
Macintosh as an example of a consistent interface for various 
applications. "...Systems that use consistent interfaces had significant 
reductions m training time" (Shaw, 1991, p. 158). The reduction in 
training time may apply, however, only to users who are already 
familiar with a GUI or a standard Mac software application. 

The term "user-friendly" is used to describe an interface that is 
easy to use. Nicholls, et al. (1990) remind us that "'user-friendliness' 
depends on who the users are; it does not necessarily mean easy for 
novices to use. An interface should be appropriate for the people 
who will be using it" (p. 43). Helgerson (1986) predicts that "user- 
friendly" will be replaced by "intuitive" to describe software that is 



self-explanatory (p. 73). Whatever it may be called, the literature 
does contain some common characteristics of a good interface. 
Zink (1991) gives the following list: 

• uncluttered items 

• knowledge of audience 

• effective use of color, spacing, windowing 

• standardization of terminology 

• key information in prominent and consistent locations, 
and 

• elimination of lengthy tutorials, documention or help 
required to use the system (p. 17). 

In the case where different types of users might use a system, 
a choice of interface may be an appropriate option. The needs of 
novice users differ from those of expert operators or of occasional, 
but trained, users. "Novices need simple, logically organized, and 
well-labeled displays that guide their actions. Expert operators do 
not need extensive labels on fields; subtle highlighting or positional 
presentation is sufficient. Display formats must be tested with users 
for comprehensibility" (Shneiderman, 1987, p. 71). 

Shneiderman (1987) describes eight underlying principles of 
good interface design: 

1. Strive for consistency 

2. Enable frequent users to use shortcuts 

3. Offer informative feedback without confusing error 
messages 

4. Design dialogs to yield closure giving users a sense of 
accomplishment and indicating the time to prepare their 
next actions 

5. Offer simple error handling, telling the user how to 
correct the error that occurred, not just that it occurred 

6. Permit easy reversal of actions 
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7. Support internal locus of control: "make users the 
initiators of actions rather than the responders" 

8. Reduce short-term memory load by using the screen 
to display frequently needed information (pp. 60-62). 

From the designers' point of view, it is clear that the first rule 

is to know the user. In Herther's review (1986) of access software 

for CD-ROM, the list of user expectations both mirrors some of 

Shneiderman's suggestions for designers and augments his list with 

items that are more specifically directed at bibliographic retrieval: 

1. Software must be easy to leam and use, even for the 
novice. Screen displays should be clear and uncluttered 
and directions straight-forward. User aids and 
instructions should be clear and meaningful to the end- 
user. 

2. Commands should be clear, simple, and use a 
minimum of keystrokes. 

3. The systems must allow for browsing. 

4. Standard Boolean searching should be employed in 
addition to being able to search on specific fields. 

5. If a thesaurus or other guides are available they 
should be carefully brought into the search process as a 
clear aid to the searcher and not a distraction or source of 
confusion. 

6. Search results of a bibliographic database should yield 
a full citation. Any abbreviations used should be clear 
and meaningful (pp. 93-94). 

Shaw (1991) notes that interface design was at first no more 
than an afterthought. Now that good interface design is regarded as 
essential to the sale of a product, more attention is being paid to this 
piece of software and hardware design, but there is no consensus on 
how to measure it (p. 157). For scientific purposes, it is necessary to 
transform the vague idea of user-friendly to quantifiable criteria. 
Shneiderman (1987) provides a list of five measurable human 
factors for evaluation: 
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1. time needed to learn 

2. speed of performance 

3. rate of errors by users 

4. subjective satisfaction 

5. retention over time (pp. 14-15) 

Thrs study will focus on items three and four from Shneiderman's 
list. Even when quantified into such categories as "How many and 
what kinds of errors are made in carrying out a particular task," the 
methods used to collect this data are themselves subjects of study. 
Qualitative methods 

"Qualitative methods, as well as quantitative methods, have 
been employed in user studies since the inception of this type of 
research in the late 1940s" (Mellon, 1990, p. 136). They were 
criticized because the findings could not be generalized, using such 
small samples that they could not be representative (Mellon, 1990). 
But there is a place for these studies. 

Qualitative analysis, also referred to as "ethnography, 
naturalistic inquiry, case studies, fieldwork, field studies, and 
participant observation" (Ary, Jacobs, and Razavieh, 1990, p. 444), is 
useful for studying events where the goal is to describe what is 
happening in context. Patton (1980, pp. 40-41) describes the process 
as holistic, not just looking at bits and pieces, but at the totality of an 
event; inductive, going from specific situations to the general; and 
naturalistic, usually occurring in the field. Fidel (1984) defines the 
case study method as an attempt to "systematically develop a 
comprehensive model describing patterns of behavior" and lists 
three conditions that point to the appropriateness of its use: "(1) a 
large variety of factors and relationships are included, (2) no basic 



laws exist to determine which factors and relationships are 
important, and (3) when the factors and relationships can be directly 
observed" (p. 273). 

Qualitative methods are particularly valuable when used to 
evaluate library applications where the presence of technology is 
meant to enhance the experience of the library user. "It is 
reasonable to find naturalistic methods filtering into library research 
through user studies. Library users are human beings whose 
information needs reflect the complexity of human life" (Mellon, 
1990, p. 136). Unlike controlled experimentation, however, the 
purpose of qualitative study is to describe and, understand a 
naturally occurring phenomenon, not to predict an outcome or 
control a situation through manipulation of a variable. 

This study will make use of three data collection methods, a 
process referred to as triangulation. Patton (1980, p. 109) defines 
data triangulation simply as the use of a variety of data sources in a 
study. Its purpose is to reduce the subjectivity that is inherent in 
qualitative methods by looking at the same event several ways. 
"Each data source has strengths and weaknesses. By using a variety 
of sources and resources, the evaluator-observer can build on the 
strengths of each type of data collection while minimizing the 
weaknesses of any single approach" (Patton, 1980, p. 158). The data 
collection methods used in this study will be observation, think- 
aloud protocol, and questionnaire. It is hoped that the strengths and 
weaknesses of each method will combine to reduce bias (Connell, 
1991). 

The Direct Observation Method 
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Patton (1980) lists six advantages of direct observation: 

1. The evaluator is better able to understand the context 
within which the program operates 

2. The evaluator can directly experience the program as 
a phenomenon unto itself, thereby making the most of an 
inductive, discover-oriented approach. 

3. The evaluator has the opportunity to see things that 
may routinely escape conscious awareness among 
participants and staff. 

4. The evaluator may gain access to information that 
may otherwise be unavilable because participants might 
be unable or unwilling to talk about it, or even unaware 
of their own actions to talk about them. 

5. The evaluator is permitted to move beyond the 
selective perceptions of others. 

6. The evaluator is permitted to access personal 
knowledge and direct experience as resources to aid in 
understanding and interpreting the program being 
evaluated, (pp. 124-125) 

The disadvantage of observation is that people may respond 
differently to a situation knowing that they are being observed. 
Also, Patton (1980) concedes, the evaluator may exhibit selective 
perception, and will observe only a limited sample of total 
occurrences (p. 158). MuUins (1991) agrees that observation may 
affect a subject's performance, and notes also that people exhibit 
emotion differently, thus frustration may be difficult if not 
impossible to spot (p. 326). Observation in this study was overt, not 
covert. The observer was discretely within view of the subjects, but 
the purpose of the study was not disclosed to participants. Subjects 
knew that they were being observed, but did not know what factors 
were of interest. 
The Think-Aloud Protocol 
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The think-aloud protocol is a data collection method whereby 
subjects are asked to engage in their normal pattern of activity, but 
to verbalize their thoughts as they occur. The only response from 
the evaluator is to remind the subjects to continue to think aloud 
should they lapse into silence. This "conversation" is recorded and 
then transcribed to become a program document. As with 
observation, the presence of the evaluator, or of the tape recorder, 
may affect the result. The subjects' thoughts may be incomplete, 
inaccurate, selective, and highly uneven in quality (Patton, 1980, p. 
158). Despite these disadvantages, think-aloud protocol has become 
a standard technique for qualitative analysis. Meyers and Lytle 
(1986) have employed think-aloud protocol in their analyses of 
reading skills. "Think-aloud procedures have the potential to assess 
strategic learning behaviors used during the actual task, and they 
have been used frequently for investigations of problem solving 
skills" (p. 140). Newell and Simon (1972) describe its use in 
understanding human problem solving: 

We employ little experimental design using control 
groups of the sort so familiar in psychology. Because of 
the strong history-dependence of the phenomena under 
study, the focus on the individual, and the fact that much 
goes on within a single problem solving encounter, 
experiments of the classical sort are only rarely useful. 
Instead, it becomes essential to get enough data about 
each individual subject to identify what information he 
has and how he is processing it. This method leads, in 
conjunction with the content orientation, to emphasizing 
the use of verbal behavior as data, because of its high 
output rate. Thus, the analysis of verbal protocols is a 
typical technique for verifying theory, and in fact has 
become a sort of hall mark of the information processing 
approach, (p. 12) 



The subjects in the study were fitted with a clip-on microphone 
that allowed them to speak normally during their search session. 
The tape recorder was in full view, and several participants were 
asked to pause while the tape was turned over to continue recording. 
The full transcripts of the subjects' conversation from the use of this 
protocol is in Appendix A. 
The Questionnaire Method 

Finally, a questionnaire was used. The Questionnaire for User 
Interaction Satisfaction (QUIS) was developed by the Human- 
Computer Interface Laboratory at the University of Maryland 
specifically to measure satisfaction with the user interface of 
software (Norman, 1989). The strengths of the questionnaire format 
are the simplicity of data analysis, the ability to directly compare 
responses, and the fact that many questions can be asked in a short 
time. The weaknesses are, though, that the "respondents must fit 
their experiences and feelings into the researcher's categories; it may 
be perceived as impersonal, irrelevant, and mechanistic. It can 
distort what respondents really mean or experienced by so 
completely limiting their response choices" (Patton, 1980, p. 206). 
Mullins (1991) notes that in addition to a questionnaire's lack of 
objectivity, users tend to emphasise the negative (p. 326). 
Summary 

These three data collection methods complement each other to 
provide a holistic picture of a novice searcher using a CD-ROM on a 
Macintosh. Verbalized thoughts and non-verbal actions were 
combined with answers from response categories to analyze the 
user's experience with an interface, both software and hardware. 



SilverPlatter's offering of ERIC on the Mac presents an interesting 
opportunity to study the Mac user interface of a CD-ROM 
bibliographic database. 
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Research Qiiftstion 

As this research is qualitative, no hypothesis was tested. 
Instead, the focus is on an in-depth understanding of the particular 
situation when novice users search the ERIC CD-ROM on the 
Macintosh. The process is "similar to librarianship itself-organizing 
knowledge to make it accessible. Naturalistic researchers make 
sense out of their data by identifying themes, arranging the themes 
into patterns of behavior, and developing statements to explain the 
patterns they observe" (Mellon, 1990, p. 69). 

The research question, then, is: What is the experience of a 
novice user when searching SilverPlatter's ERIC CD-ROM on the 
Macintosh? 



Definition of Terms 

CD-ROM Product: a software program and/or 
database that resides on a cor.ipact-disc-read-only- 
memory medium, a high-density storage device 
capable of integrating text, data, graphics, audio, and 
video. 

Command-line Interface: a type of interaction with 
computer software requiring the user to input specific 
words and/or to press specific keys on the keyboard 
in order for the request to be processed. 

ERIC: Educational Resources Information Center, 

education database funded by the National Institute of 
Education (NIE) and available in paper, on CD-ROM, 
and through online services. Contains bibliographic 
citations from Current Index to Journals in Education 
(CUE), Resources in Education (RIE), and Exceptional 
Child Education Resources (ECER). 

GUI: graphic (or graphical) user interface. A type of 
interaction with computer software comprising 
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windows, icons, menus, and a pointing device (usually 
a mouse). 

Icon: a graphic representation of a command or process, 
such as a '?' to represent help or a picture of a house 
to represent a main menu (or home screen). 

Interface: "what the user sees, hears, and touches in 
interacting with a computer system" (Shaw, 1991, p. 
155). 

Menu-Based Natural Language: a type of interaction 
with computer software where words and phrases are 
displayed so the user can point to them to select them 
(Hendrickson, 1989, p. 217). 

Novice: a category used in this study to indicate the 
subject's level of exf)erience using a computer. Note 
that this categorization does not apply to experience 
searching ERIC. None of the subjects in this study had 
searched ERIC on the Mac, though some did have 
experience wiiu ERIC through other sources. 

Qualitative Research: a process comprised of describing 
and categorizing behavior, useful for studying events 
in context. 

Think-aloud Protocol: a data collection method where 
subjects are asked to engage in their normal pattern of 
activity, but to verbalize their thoughts as they occur 
(Patton, 1980, p. 158). 

Triangulation: the use of a variety of data sources in 
order to reduce subjectivity (Patton, 1980, p. 109). 



Methods and Results 
This study employs qualitative techniques to document the 
experience of novice users when searching SilverPlatter's ERIC CD- 
ROM on the Macintosh. 

Naturalistic inquiry focuses on i^-depth understanding of 
a single situation, not on results to be generalized to all 
similar situations. Naturalistic researchers cannot make 
the broad claim of the quantitative research: "Given this 
situation, here is what is likely to happen." They must be 

Tu"^ I'oof^' ^his setting" 

(Mellon, 1991, p. 18). ^ 

Sample 

Ten subjects were selected from a group of volunteers from two 
sections of a Foundations of Librarianship course at Kent State 
University's Columbus extension program. This is a nonprobability. 
purposive sample where this class is judged to be typical (Aiy, et al., 
p. 177). The small sample size is typical of qualitative studies where 
a large volume of data will be generated and findings should not be 
generalized. 

The only requirement met by the volunteers was no previous 
experience searching the ERIC CD-ROM on the Macintosh. A profile of 
subjects was obtained vi. questionnaire as detailed in Tables 1-6. 
Most of the subjects were female, and the average age was 33. Four 
subjects classified their experience with computers as casual, and 
four as experienced, with novice and expert categories claimed by 
one each. Six subjects indicated a preference for IBM or compatible 
personal computers, three for Macintosh. One subject had no 
preference. Only one subject had no searching experience, eight had 
searched online public access catalogs (OPACs), while seven had 



searching experience with CD-ROM. Five had searched DIALOG or 
another search service. Table 1 details subjects' experience with 
devices^ hardware and software. Notably, two subjects had no 
experience with a mouse, and four subjects had never used a 
Macintosh. 
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Table 1 

Description of Subjects 



Gender 



Male 




rcmaiw 


1 


Age 




24 




9 7 


Z 




1 








1 


47 


1 
1 






IN UVICC 


1 
1 


■ ^ O C 11 o 1 


A 


nxpcriwuccu 


A 

4 




1 

I 






loiYi or compaiiDic 


O 




D 




1 
J 


Rxneriencft' Sparrhin o 






5 


CD-ROM 


7 




9 


DIALOG or other 


< 


scaicn sysicui 




iN unc 


1 
1 






Hardware and Software 




Keyboard 


10 


Mouse 


8 


Word processing 


10 


Computer games 


6 


IBM PC/compatible 


8 


Macintosh 


6 


Mainframe 


2 


CD-ROM drive 


6 



Procedures 

This study employs a qualitative, or naturalistic, design. A small 
sample of 10 students from two sections of Foundations of 
Librarianship was given a search request to complete using 
SilverPlatter's ERIC CD-ROM on the Macintosh. An area was set aside 
to assure some degree of privacy to the subject. The observer sat 
within view of the participant. See Figure 2 for an overhead 
schematic of the arrangement of the study area. Each session was 
scheduled to last one hour, but participants were informed that time 
was not an issue in the evaluation. 



Figure 2 

Overhead Schematic of Study Area 



O O 



Tape recorder 




Observer 



Mac 



O ^ 



Subject 



Reference 
Materials 



Using a script (Appendix B) to ensure uniformity of directions, 
the researcher explained the study. The subjects were informed that 
their session would remain confidential, and that the software was 
being judged, not their performance. They were advised that they 
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could stop the session at any time if they felt uncomfortable. The 
think-aloud protocol was explained, as well as the task and the 
equipment. Materials on hand included the SilverPlatter manuals 
(Getting Started manual and User Reference manual in one binder), 
an ERIC thesaurus, and the MacSPIRS Quick Reference Card. The CD- 
ROM and computer were ready for use. Participants were 
encouraged to ask any questions before the session actually began, 
but questions during the session were discouraged. At the conclusion 
of the session, questions were again accepted. These procedures are 
recommended by Gomoll, 1990, pp, 87-90, 

The search question was presented on paper along with space 
to be used at the subject's option to jot down concepts, search terms, 
or ideas. The researcher was seated within view of the participant. 
Participant actions, use of paper sources, and non-verbal 
communication, as well as any other behaviors that proved 
significant were noted. As Patton (1980) recommends, observation 
notes also included "the observer's insights, interpretations, 
beginning analyses, and working hypotheses about what is 
happening in the setting" (p. 161), These notes were transcribed 
and integrated with the think-aloud protocol transcript, following 
procedures discussed by Patton, 1980, p. 149. 

The think-aloud protocol was recorded and later transcribed 
and coded for analysis (see Appendix A). The tape recording was 
begun when the subject was ready to begin the search. The only 
researcher intervention during the search session was to remind the 
participant to continue to think aloud and to offer minimal 
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instructions when necessary. Some examples of my intervention 
from the transcripts are as follows: 

use the mouse; move the mouse up here to this menubar, 
you can click on that, and then go down to select one of 
these topics; when you get to the end of the pad, pick that 
up and move it back down; drag across there, and hit 
backspace, will delete it; double click on MacSPIRS; come 
down to printing records, and that's a secondary menu, if 
you'll move the mouse over, and then select one of those. 

At the end of the search session, the QUIS questionnaire was 

administered. Participants were asked to make any additional 

comments in writing to close the interview. These comments were 

added to the transcripts. 

Instrumentation 
Tri angulation of data collection was employed to reduce observer 
bias. The three methods used were think-aloud protocol, systematic 
direct observation, and questionnaire. 

The search question used was identical to that used in the 
Puttapithakporn study where novice users searched SilverPlatter^s 
ERIC CD-ROM on the IBM PC: 

Design and conduct a search for the following request: I 
am concerned about obesity in young people. What is the 
current thinking on teaching young children (preschool 
and kindergarten through third grade) about nutrition 
and exercise? (S. Puttapithakporn, personal 
communication, February 19, 1 992). 

Appendix B contains a script which was read to each subject before 

the study began. This script briefly explained the methodology of 

the study, explained the think-aloud protocol, and introduced the 

questionnaire. 



Think-aloud protocol 

The subjects were asked to think aloud during their search, and 
this was recorded on audio tape for later transcription and coding 
(Appendix A). 
Observation 

The author, within view of the subject, observed the search 
session, ^nd those notes are integrated in the transcripts in italics. 
Questionnaire 

Finally, the Questionnaire for User Interface Satisfaction (QUIS) 
was administered with some modifications to the subject description 
section and some on;issions from the user reaction section, QUIS was 
licensed from the Human-Computer Interaction Laboratory, 
University of Maryland at College Park (See Appendix C for the Site 
Software License Agreement,) Comments written at the end of the 
questionnaire were added to the transcription from each search 
session. 

Presentation and Analysis of Data 
Transcripts and author observations were integrated and coded 
according to the scheme in Table 2 with occurrence totals from all 
searchers. Data from the transcripts was broken down into three 
major categories: reference materials used, interface issues, and 
problems. Questionnaire responses were used to further clarify the 
data, A detailed list of events for each of the categories is listed in 
Appendix D with reference to the codes found in the transcripts. 



Table 2 




Total Coded Events bv Categorv 




Reference Materials Used 




Quick reference card 


1 1 


User manual and Reference guide (binder) 


6 


Thesaurus 


15 


Help screens 


20 


? icon (context-sensitive help) 


8 


Introductory screen, did subject read 


5 Yes/5 No 


Macintosh instructions given by author 


8 


Index, online 


2 


Exploration of software features 


7 


Interface issues 




System plusses 


18 


System minuses 


12 


Intuitive navigation 


13 


Navigational errors 


13 


Mouse problems 


18 


Problems 




Errors 


48 


''I don't understand what this means" 


26 


Logic errors 


14 


Response time 


13 



Reference Materials Used 

Each of the subjects used reference materials of some kind 
whether hardcopy or within the software. Some quotations from the 
transcripts show some of the varying attitudes about use of 
reference materials: 

I think if I was going to do this I would have read the 
instructions before I started. This is what I mean where 
I sometimes, I just go for the obvious and then after that 
I'm not sure, you know? Uh, that's why I don't think I 
utilize it the way, you know, completely. If I were 
paying for it I would have planned the search ahead of 
time. I also would have spent some time looking through 
the hardcopy manuals. 



The noted events do not include intentions, only times the 
sources were actually consulted. Each type of reference material will 
be discussed in turn. 

Quick reference card . Of the eleven uses of this reference tool, 
three (27%) were to check for an explanation of the ERIC 
abbreviations. Two subjects just looked at it to see what it was. Four 
subjects did not look at it at all. The one novice user accounted for 
the most uses. That user checked twice for an explanation of 
abbreviations and once for information on whether ERIC contained 
actual articles or just citations. Those who identified themselves as 
having a preference for the Macintosh consulted the quick reference 
card most frequently. See Table 3 for the breakdown of events by 
both experience and computer preference of subjects. 

The reference card supplied here was actually produced for the 
IBM product. Even so, it provided useful information for those who 
consulted it for abbreviations, limiting of fields, truncation, and help 
with Boolean operators. This is a comfort device, as well as an 
informational one. This sa.ne information was available through help 
screens or context-sensitive help, yet the one novice user obviously 
felt more comfortable consulting a hardcopy source than online help 
(3 times vs. 1 time). 



Table 3 

Use of Quick Reference Card by Experience and Computer Preference 







General 


experience 








corr : 


= -0.535 




corr = 0.500 


Events 


Novice 


Casual 


Exp 


Expert 


IBM Mac None 


0 


0 


2 


1 


1 


3 1 0 


1 


0 


0 


2 


0 


2 0 0 


2 


0 


2 


1 


0 


1 1 1 


3 


1 


0 


0 


0 


0 1 0 


TOTAL 


3 


4 


4 


0 


4 5 2 



User manual and Reference guide (binder V These two tools 
were combined into a thick binder. There were a total of six events 
by four subjects. Four uses (67%) were just to look at it. The other 
two events were to look for specific information, but no use was 
made of either a table of contents or an index. The subject with no 
computer preference had the most events (see Table 4). This binder 
explained software use and all interface features, many of which 
would have aided the subjects significantly. Some interface features 
might have been used had the subjects been aware of them. The size 
of the user and reference manuals may have deterred their use, or it 
may have been due to the contrived nature of the situation. One 
subject opened the manual, looked at the thickness of it and 
immediately closed it. 
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Table 4 

Use of User Manual bv Computer Preference 
correlation = 0.813 



Occurrences 



IBM 



Mac 



None 



0 
1 

3 

TOTAL 



5 
1 
0 
1 



1 
2 
0 

2 



0 
0 

1 

3 



Thesaurus. Of the fifteen events of thesaurus use, only one was 
just to see what it was, and that by the subject with no computer 
preference. Four subjects (27% of total events) checked for 
descriptors before searching. Nine times (60% of total events) 
descriptors were checked by five subjects while a search was in 
progress. This was usually in preparation for the next search, but it 
also occurred during long search times when the subject appeared to 
be passing the time. Only one subject, the expert computer user, 
checked the thesaurus for age-level descriptors. Experienced 
searchers used the thesaurus most frequently, as shown in Table 5. 

Thesaurus use caused an interesting problem. When the 
descriptor contained two separate words, such as "young children" or 
"grade one," it was listed just like that. The software, however, 
would then search the words separately, "young" and "children," 
causing long search times, and in one case, error messages indicating 
the search was too complex to be performed. If the subjects had 
used the online index, they would have seen that descriptors with 
multiple words were recognized in the software by a hyphen. See 
the discussion of index use on page 38. 
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Table 5 

Use of Thesaurus by Experience 
correlation = 0.610 



Occurrences 


Novice 


Casual 


Experienced 


Expert 


0 


1 


1 


0 


0 


1 


0 


2 


2 


0 


2 


0 


0 


I 


0 


3 


0 


1 


1 


1 


TOTAL 


0 


5 


7 


3 



Help screens . Two subjects never consulted the help screens. 
One subject, who had no experience with the Mac, consulted them for 
nine events out of the total twenty (45%). Once this searcher found 
the help screens, she used them efficiently to guide her through the 
processes in question before she even attempted them on her own. 
Though the twenty events were for a variety of questions, fifteen of 
those events (75%) ended with no answer being found. Either the 
subject chose an incorrect topic, looked for information that was not 
there, or needed a very specific answer that they could not find. The 
transcript quotations below illustrate the problems subjects had 
when they selected incorrect topics, failed to select a secondary 
menu, or just got frustrated with help: 

And it's telling me what's contained in ERIC, and Fm just 
clicking because it's giving me lots of information that Tm 
not interested in. OK, so this is telling me how to, um, the 
terminology to use, I'm still unsure how I move from 
where I am now, into something narrower. I'm still real 
unsure how to narrow the search down. Fm going to go 
up to help to see if it can tell me something about 
printing records, no, it doesn't really say anything. I was 
looking through the fields to see if there's a way to 
expand the fields, and there doesn't appear to be, do, let's 
start searching and see what happens. 
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Two issues caused ithe most frequent references to help 
screens. Eight of the twenty events (40%) were subjects trying to 
find out how to return to the search screen from viewing records 
found. See more on this problem under the discussion of 
navigational errors, page 40. Sc?ven of the twenty events (35%) 
concerned how to print records (see the discussion of this under 
errors, page 41). 

? icon (context-sensitive help) . Four subjects never tried this 
feature. One event, counted as an error and not an event here, was 
the use of the question mark key on the keyboard instead of clicking 
the icon. Three of the eight (38%) involved the searcher looking for 
information on how to print selected records. Only one subject 
actually moved the question mark to the field to reveal the correct 
procedure. This was the only event of the eight (13%) that yielded a 
successful response. Seven of the eight events (88%) were 
performed by those who expressed a preference for IBM or 
compatibles (see Table 6). 

The lack of appropriate use of this feature highlighted an 
intuitive feature of the interface, however. When the ? icon was 
clicked, the normal arrow of the cursor changed to a question mark. 
Those users who did not understand context-sensitive help, did 
notice the cursor change and intuitively clicked again to return to the 
mouse. The one subject who did not do this immediately discovered 
the ability to use context-sensitive help when he received help on 
the help menu. He then clicked the icon to return to the mouse. 
Toggle functions were well understood, on the whole (see the 
discussion of system plusses, page 39), 
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Table 6 

Use of ? Icon bv Computer Preference 
conelation = 0.598 



Occurrences 



IBM 



Mac 



None 



0 
1 

2 

TOTAL 



1 
3 
2 
7 



2 
1 
0 
1 



1 
0 
0 
0 



Introductory screen . Five of the ten subjects (50%) read this 
information, five did not. Those who did read it did so thoroughly 
and patiently. All three of the male subjects read the introductory 
screen while only two of the seven females did (correlation of 0.655 
between the reading of the introductory screen and gender). 

It would be interesting to see if arranging information on this 
screen in an outline format as opposed to straight text would 
increase the percentage of female searchers who read it. This would 
be an excellent place to provide a brief indication of what the 
database contained, the dates included, or searching tips. Copyright 
information could be displayed on the screen as the product is 
loading and then give way to a more useful introductory screen. 

Macintosh instructions . The researcher interrupted eight times 
to give Mac instructions to a total of four subjects. Five of the events 
(63%) were to one subject who had no Mac experience. All eight 
events occuned with female subjects, two of whom had no Mac 
experience, and one who stated she had used it "maybe 3 times". Of 
the three males in the study, only one had no Mac experience, but he 
had extensive Windows experience which appeared to carry over 
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well to the Macintosh, Seven of the eight events (88%) of Mac 
instruction went to three of the six subjects who identified 
themselves as having a preference for IBM or compatibles. 

With some basic Mac assistance, even those who had never 
used a Mac before were able to successfully navigate through this 
basic search task* The problem for institutions then, is how to 
provide this basic instruction to those with no Mac experience. 

Index> online . The online index was used by only two subjects, 
one time each. Both events occurred while the subject was searching 
for a way to narrow a search, and neither found its use helpful. One 
subject entered the main search request, with Boolean operators, im 
the dialog box for the index. One subject clicked on several topics, 
unknowingly transferring them to the search dialog box which later 
caused confusion. 

Several of the other eight subjects mentioned an online index, 
some even looked for it. If the index had been more prominently 
displayed, with a button perhaps, instead of buried on the right of 
the menubar, there might have been greater use of it. This quotation 
from the transcript is illustrative: 

That's actually something that, something else that I 
could have done, is checked the online thesaurus, and I'm 
not sure if there is one. Online thesaurus meaning terms 
that are indexed. Uh, I haven't seen any mention of that, 
I think ril just continue to look through the hardcopy. 

Exploration of software features . Seven subjects made no 

attempt to explore buttons or menubar items of the software. Of the 

seven total events, four of them (57%) occurred while the subject 

tried to return to the search screen from viewing records, and none 
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of these were successful. Two subjects accounted for three events 
(43%) where they simply expressed interest in seeing other parts of 
the software. Subjects who explored features were those less 
experienced (see Table 7). The subject with no computer preference 
accounted for four (57%) of the events (correlation of 0.764 between 
the exploration of software features and computer preference). 



Table 7 

Exploration of Software Features bv Experience 
correlation = -0.538 



Occurrences 


Novice 


Casual 


Experienced 


Expert 


0 


0 


2 


4 


1 


1 


0 


1 


0 


0 


2 


1 


0 


0 


0 


4 


0 


1 


0 


0 


TOTAL 


2 


5 


0 


0 



Interface issues 

System plusses . Of the eighteen events recorded, ten (56%) 
involved the interchangeability of pushing the enter (or return) key 
and clicking on the start search button, and the ability to type or 
select the Boolean operators. The inclusion of" basic IBM interface 
elements such as using the enter or return key was a plus for those 
with IBM experience. Most of these subjects used both the button 
and the return key interchangeably without even being conscious of 
it. The ability to click a button to add Boolean operators to the 
search string was awkward because the hand had to be lifted from 
the keyboard during typing. It did, however, keep the operators 
within sight of the user to serve as a reminder. One subject took 
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their display on the screen as a sign that the search software did use 
Boolean logic. This is one of the advantages of a GUI. 

Svstgm minuses. The events recorded as system minuses were 
varied, but four of the total twelve (33%) involved a system beep 
with no explanation. The three male subjects accounted for 58% of 
the events. Eighty-three percent of the events were recorded from 
those who expressed preference for IBM or compatibles. See Table 8 
for detail. 

One experienced searcher who received several errors 
regarding the complexity of his search held his ground, as shown in 
this transcript quotation. His search used several two-word terms 
(example, first grade) anded together which caused the messages; 
however the system gave no clue that hyphenation of those words 
would have caused it to search them as one term. The message could 
have indicated that checking the online index (and instruction on 
how to locate it) could help reduce the complexity of a search. That 
is where the hyphenation of multi-word terms was obvious, whereas 
the thesaurus was misleading: 

Um, the question I always have in doing these searches 
and receiving errors is asking if I've done something 
wrong. Um, in this case, I don't think that I have. 



Table 8 

System Minus Event s bv Gender and Computer Preference 





Gender 


Computer preference 




corr = 


-0.636 




corr = -0.511 


Events 


Male 


Female 


IBM 


Mac None 


0 


0 


3 


1 


1 1 


1 


1 


3 


2 


2 0 


2 


1 


1 


2 


0 0 


4 


1 


0 


1 


0 0 


TOTAL 


7 


5 


10 


2 0 



/ 

Intuitive navigation . These thirteen events were noted 
because the subject seemed to be new to these functions, yet readily 
used them correctly. Functions included the toggle of the "?" icon 
and radio buttons^ use of the close window box, and use of the 
scrolling arrows to move up and down. For a complete list, see 
Appendix D. The majority of events were noted for those without 
Mac or mouse experience, as shown in Table 9. 

These functions are worth emphasis, because this is where the 
image that the designer had is transferred to the user's model of the 
system. Toggle keys and arrows appear to make sense to these 
users. It is also worth noting that one event was the use of the close 
window box to return to the search window. This was used by only 
one of two subjects who did not have a navigational error on this 
point. See the following discussion of navigational errors. 
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Table 9 

Intuitive Navigation bv Mouse and Mac Experience 



Mouse experience 
corr = -0.674 



Mac experience 
corr = -0.591 



Occurrence 



No Yes 



No Yes 



0 
1 
2 
7 

TOTAL 



0 4 

1 3 

0 1 

1 0 
8 5 



0 4 
2 2 

1 0 
1 0 
1 1 2 



Na V i g a tion a 1 errors . This subset of system minuses was so 
pervasive that it was analyzed separately. This error occurred when 
a search had been completed and the records found were shown. But 
eight of ten subjects experienced trouble getting back to the search 
dialog box. Ideally, the subject had a choice of clicking within the 
search window, closing the records found box, or selecting Find 
Records (or 38 F) from the a menu on the menubar. 

There were a total of thirteen events. Two subjects accounted 
for three events (23%) when, after trying numerous options, they 
accidentally clicked in the search window. The most frequently 
occurring answer was to restart the session (6 events by 3 subjects, 
46% of total events), three of those after some exploration of the 
menubar, including seeing the Find Records option. One subject quit 
the session and had to restart MacSPIRS. These quotes from the 
transcripts show what a confusing and frustrating task this was: 

Fm sure there's an easier way to do this. I'm going to 
start over, how can I do this? Oh, show subset, that 
might tell me how to get back. Fve failed miserably. 
How do I get back to where I was, previous record? How 
about select fields again and see what happens. Or do 



you have to print? OK, so Tm thinking here, should I get 
help? 

This event was so pervasive, that it is interesting to see why 
two subjects did not experience it. There was no one distinguishing 
characteristic to separate these subjects from the rest of the study, 
although they did appear to be the most experienced. One, however, 
had no Mac experience at all. Both subjects were male (of three 
males in the study). Both had extensive searching experience with 
CD-ROM and online services, primarily DIALOG. One subject, 
identified as the expert computer user, refined his search so 
completely before displaying records that he did not need to go back 
to the search window. After his session, I asked him to return to it, 
and he clicked the close window box without hesitation. The subject 
with no Mac experience did have extensive experience with 
Microsoft Windows. These two users clearly had a different model of 
the system than the others. In any event, this basic task could be 
made more noticeable. 

Mouse problems . One subject had fourteen (78%) of the 
eighteen recorded errors of mouse movement. This was the only 
subject who had experience with neither the Mac nor computer 
games. Even this subject gained confidence and improved usage 
during the short search task. She began the session ignoring the 
mouse, then when she started to use it, her entire body moved along 
with it. By the end of the session she was more at ease, and mouse 
movement was less of an effort for her. The only other subject 
without mouse experience experienced none of these errors (see 
Table 10). For three subjects with mouse experience, the primary 
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difficulty seemed to be accessing secondary menus under the help 
menu. One other subject exhibited initial hesitancy when first using 
the mouse which was quickly overcome. These transcript quotations 
illustrate the problems these subjects experienced: 

Oops, OK, I need to hold the little clicker down apparently 
in order to have those items show. There seems to be a 
narrow band that one needs to move the mouse in in 
order to get the assistance, and that's a little tricky. I 
don't know how to get in there because I have held the 
clicker down to get over th^.re and when I release, hm, 
OK, now it's all right. I am having difficulty learning to 
use this mouse properly. That's exactly what I need if I 
can get over there. Oops, I need to go up one. 



Table 10 

Mouse Problems by Mouse Experience 
correlation = -0.635 



Occurrences 


No mouse exp 


Mouse exp 


0 


1 


4 


1 


0 


4 


14 


1 


0 


TOTAL 


14 


4 



Problems 

Errors . The majority of errors were made made by those 
subjects without any previous CD-ROM experience (see Table 11). 

Fourteen of the forty-eight errors (29%) concerned printing 
selected records* Three subjects accomplished t iis by using the 
LaserWriter dialog box to print from page x to page y entering record 
numbers instead of page numbers. Another subject started printing 
from a record number in the print dialog box and then let printing 
continue through all records. Three subjects printed all records 



without realizing it. One subject retrieved five applicable records 
and printed all records, eliminating the need to select specific 
records. (This was also the only subject who had no errors.) There 
was no correlation between users who had difficulty printing 
selected records and their level of experience or their experience 
with IBM or Mac computers. This function could have been more 
obvious. Some quotations from the transcripts illustrate some of the 
problems encountered: 

I thought it would print the one that was on the screen at 
the time. OK, urn, what Td do at this point, I can't figure 
out how to print the one record, and I don't feel like 
spending any more time on this, so I'm just going to print 
both and manually separate the two. 



Table 11 

Errors by CD-ROM Experience 
correlation = -0.693 



Occurrences 


No CD-ROM exp 


CD-ROM exp 


0 


0 


1 


2 


0 


1 


3 


1 


3 


4 


0 


2 


9 


1 


0 


17 


1 


0 


TOTAL 


29 


19 



"I don't understand what this means" . These problems were 
usually identified by exactly this phrase. The subject was confronted 
with terminology or a procedure with which he or she was not 
familiar. Examples include the following: click on any word or drag 
across any phrase, unsure whether or not it printed, abbreviations 
not obvious, and records formatted so far when printing. Other 



events were more general, such as the help text to narrow a search, 
and whether or not the search was complete. One subject noted 
twice that it was just not simple enough to perform a search. See 
Appendix D for a complete list. 

Males accounted for 17 of the 26 total occurrences of "I don't 
understand" events (65%, for a correlation of -0.569), though 12 of 
those were by one person, the only subject to identify himself as a 
novice computer user. Generally, the more experienced the user, the 
fewer of these events were recorded (see Table 12). 

Subjects were quick to blame themselves for many of these 
events, as evidenced in these quotations from the transcripts: 

But no records were formatted so far, so I did something 
wrong there. Each line begins with some kind of code or 
abbreviation that, um, sometimes it's just not obvious 
what it means. I wonder what this records formatting 
this is all about. Oh gosh, I'm really not used to this Mac, 
how to do it, I don't knoN^'^ Well, I have no idea what just 
happened. It was showing the percentage searched and 
then all of a sudden there was nothing on there. I may 
have done something with the mouse. Probably did. So, 
I guess all the blank means is that it finished the search, 
and I just wasn't paying attention. 



Table 12 

"I Don't Understand" F. vents bv Various Experiencg 





General experience 


level 


CD-ROM cxp 


Searching exp 






corr = 


-0.633 




corr = 


-0.569 


corr = 


0.888 


Occ. 


Nov. 


Casual 


Exp. 




No 


Yes 


No 


Yes 


0 


0 


1 


2 


0 


1 


2 


0 


3 


1 


0 


1 


1 


1 


0 


3 


0 


3 


2 


0 


1 


0 


0 


0 


1 


0 


1 


4 


0 


0 


1 


0 


0 


1 


0 


1 


5 


0 


1 


0 


0 


1 


0 


0 


1 


12 


1 


0 


0 


0 


1 


0 


1 


0 


TOT 


12 


8 


5 


1 


17 


9 


12 


14 



Logic errors. Although errors in search logic are outside the 
scope of thi study, fourteen events were coded as such. One subject 
accounted for five of these (36%) by combining two searches, one of 
which had zero hits. The two subjects who had no experience with 
OPAC's accounted for half of the events (correlation of -0.583). 

Response timg. Slow response time was enough of an issue to 
warrant comment or explicit action for a total of thirteen events. 
Four of those (31%) were extra mouse clicks by two subjects due to 
impatience. Six events (46%) were noted as obvious frustration by 
three subjects. Seven events (54%) were from the three males in the 
study (correlation of -0.570). 

As would be expected, more events were noted by those with 
computer game experience, but lacking experience with OPAC's, OCLC, 
or mainframes (see Table 13). Curiously, the correlation between 
DIALOG experience and these response time events was very low 
(conelation of 0.084). 



The following transcript quotations show some frustration and 
criticism of both the software and the user's personality: 

I'm used to having things come up automatically. I'm not 
used to waiting until a search is over 25% complete to 
have it come back and tell me that there's anything in 
there. This isn't a good one for an impatient person. This 
is CD-ROM, this is here. I'm not paying online charges or 
anything so it can sit there and grind, and it costs the 
same to sit there and grind as not. It makes me reluctant 
to do the logical last step of this search. If this 
application was a little bit faster and a little bit less 
awkward, I would probably be tempted to experiment 
around with some of the search terms. . . .It really doesn't 
encourage me to play around with the database at all. 
I'm angry because it told me that after it had searched 
for five minutes. So, the assumption is, I've got this client 
that's waiting on me to answer this question, and I've 
been messing around with this for half an hour, and I 
can't go to the client and, say, look, I don't like this. I 
think my search efficiency goes down after a certain 
time. 



Table 13 

Response Time Event s bv Various Experience 





OPAC 


exp 


OCLCexp 


Cgame exp 




corr = - 


■0.716 


corr = 


-0.547 


corr = 0.722 


Occ 


No 


Yes 


No 


Yes 


No Yes 


0 


0 


4 


2 


2 


3 1 


1 


0 


1 


1 


0 


1 0 


2 


0 


3 


3 


0 


0 3 


3 


2 


0 


2 


0 


0 2 


TOTAL 


6 


7 


13 


0 


1 12 



Questionnaire for User Interface Satis faction (OTITS) 

The questionnaire contained seven parts: 
Part 1: Past Experience 
Part 2: Overall User Reaction 



Part 3: Screen 

Part 4: Terminology/Messages 

Part 5: Learning 

Part 6: System Capabilities 

Part 7: User Comments 

The subjects' past experiences (Part 1) were used to further 
describe the ratings they gave where a significant correlation 
(greater than 0.50 or less than -0.50) existed. Responses to Parts 2 
through 6 were given on a Likert scale. Part 7, User Comments, was 
appended to the respective transcripts and included in that analysis. 

Issues were evaluated on a Likert scale from one to nine with a 
"not applicable" choice also available. The lower the number circled 
the more negative the user's response to the interface, and the 
higher the number circled the more positive the response. A score of 
five was considered neutral. High scores indicated that the user 
perceived the interface positively: satisfying, consistent, clear, and 
fast enough. Low scores indicated that the user probably had a 
different model of the interface than the designers intended: 
terrible, confusing, inconsistent, and too slow. 

Overall User Reaction. The average marks for this section were 
neutral. In general, those who preferred IBM PC's and compatibles 
gave lower marks than those with a Mac preference. The highest 
scores were from those with a Mac preference. Dialog users also 
tended to give lower marks than those who had no Dialog experience. 
Those with mainframe experience were more likely to describe their 
interaction as stimulating as opposed to those without mainframe 
experience, but also classified the interface as rigid. 
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Sfilfifia. The general response to this section was positive; the 
average score for each question was six or higher. The highest 
ratings were given by those who had used IBM PC's and compatibles. 
The GUI apparently was not an obstacle to their task, and their 
favorable ratings suggest that the characters and screen layout were 
consistent with their logic. The subjects who had Mac experience 
appeared to find the screen sequencing less confusing than those 
without. 

Terminology/Messsages. The general response to this section 
was positive, with the exception of a question on the helpfulness of 
error messages where the average response was 3.8, the lowest 
average score on the questionnaire. Only 20% of the subjects gave 
error messages a positive response with two subjects neutral, and 
60% with negative responses. This is consistent with the discussion 
of system minuses earlier, where the major problems cited were 
system beeps with no explanation or messages that the subjects did 
not understand. 

Although most subjects found the use of terminology consistent 
and familiar, the one subject who had no experience searching found 
the terminology confusing and felt that the system did not keep him 
informed about what it was doing. All those with IBM experience 
responded positively about the computer keeping them informed. 
Generally, the novice user gave the lowest marks, and the expert 
user gave the highest marks on this section. 

Learhing. The average responses to this section were neutral. 
Generally, the lower the experience level of the user (novice, casual, 
experienced, expert) the more difficult the system was to learn. The 
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most favorable responses were from the expert user. Those with 
OPAC experience tended to be more positive about learning to 
operate the system and the clarity of help messages. Users with CD- 
ROM experience were more positive about tasks being straight- 
forward to perform, while those without previous CD-ROM 
experience were neutral or negative. Most of those who had 
experience with CD-ROMs (5 of 7. 71%) gave high marks for ease of 
learning. Of the five subjects who had experience with DIALOG, four 
were positive about the ease of exploration in the system, one was 
neutral. 

It is interesting to note here that all subjects successfully 
completed the search task assigned, regardless of their level of 
experience. The time it took and the frustration level they exhibited 
varied greatly, but even in this contrived situation, they were able to 
print out relevant records from the ERIC database. The low marks 
for learning may have come from that frustration level rather than 
from a user's inability to figure something out. 

System CapabiUtigs. The interface received negative responses 
on average concerning system speed and consideration of 
experienced versus inexperienced users. There was a positive 
average response concerning the ability to correct mistakes. Males 
and those with experience using Dialog or mainframes were most 
likely to give low ratings for system speed. Only two of the seven 
females gave negative responses here. Certainly online applications 
with high-speed lines and large mainframes offer some advantages 
in terms of speed, and, as evidenced in the transcript comments 



concerning response time, these users were aware of those 
differences. 

Males also were unanimously negative about the interface 
taking into account the experience level of users. This is interesting 
because one of the men was quite inexperienced while the other two 
had significant experience. None of them were satisfied with the 
level of interaction provided. Five of the six users who preferred 
IBM PC's and compatibles gave negative responses here, while those 
with a Mac preference were split among negative, neutral, and a 
positive response. Of the five with Dialog experience, four were 
negative. One of these users had asked for a native mode before 
even attempting to perform the search with the interface presented, 
preferring a blank screen from which to enter search terms with 
Boolean operators by way of commands. 
Summary 

It would appear that the standard Mac interface as 
implemented in this product did not win over any users with a 
preference for IBM PC's or compatibles. Users at all experience 
levels had problems with the interaction provided, novice users 
expressing a need for more direction and clarity, experienced users 
looking for a more direct way to conduct their search. The most 
glaring negative concerned the error messages provided. 



Implications and Applications of Results 
Although the findings from the study of such a small group cai 
not be generalized, the experiences of these searchers contributes to 



our understanding of CD-ROM use and could be well used in future 
interface design. 

Implications 

A better understanding of how users interact with CD-ROM 
search software allows libraries to develop more effective training 
materials and to focus collection development efforts on well- 
designed products. 

Users without Mac experience needed some basic instructions 
to allow them to complete their search. Installations without both 
IBM and Mac platforms available would need to make some 
provision for this information, whether a small reference handout or 
an optional online tutorial presented to the user before the 
application is begun. Where dual platforms are an option, one would 
assume that ERIC on the Mac would be chosen by those with a 
preference for, and thus experience with, the Mac. The Mac is 
advertised, though, as easier to use, and novice users may find 
themselves frustrated with such basic processes as moving the 
mouse and closing a window. Support staff or nearby librarians who 
may be called upon for assistance should be prepared. 

Subjects used the quick reference card frequently, even when 
it was an inappropriate reference tool. Local development of such a 
card might include answers to most frequently asked questions or 
searching tips, such as not to search for the word "education" in ERIC 
or the usefulness of the online index. Leave the binder with user 
and reference manuals behind the desk for the staff to use during 
training and when answering questions. Its size was clearly a turn- 
off to users with a task to complete. 



As for the thesaurus, while its use was heavy, the subjects 
would have been better served had they used the online index. The 
index showed the number of hits for each word or phrase, and 
subjects made up of two words were hyphenated, reducing search 
time significantly. Perhaps if the thesaurus had not been available, 
more subjects would have looked for, and used, the online index. 

In general, Mac users gave this product high marks, and it is a 
step toward providing these users access to the wealth of 
information available on CD-ROM. Some design changes, based on the 
experiences of these searchers, are suggested below. 

Appligati^na 

Kay describes the computer as a medium of expression, just as 
the book. To understand what is being communicated, the medium 
must be removed. As a book is being read, the reader does not focus 
on the act of reading, but is interacting with the material being read. 
Such is the task for the interface designer (Kay, 1990, pp. 192-3). 

Authors agree that users should be an integral part of interface 
design. Gomoll (1990) includes users in all phases: brainstorming, 
prototyping, building, and evaluating (p. 85). Zink (1991) maintains 
that involvement of the user should come "early in the software 
design process" (p. 17). But software interface design is an ongoing 
process, and this study points out some opportunities for change. 

The most frequently occurring problem was in returning to the 
find records box from viewing records. Even those who happened 
upon the item "Find Records" under the menubar failed to 
understand it or use it. This function could be made more obvious 



by the use of a button, labelled perhaps "Search Again" or "Refine 
Search," at the bottom of the viewing records display. 

Printing selected records was another problem area. A change 
to ease this function would be to reword the "Select Record" button to 
read "Select to Print." Even though this button does not cause ihe 
record to be selected, it at least provides the help text needed to 
inform the user of the correct procedure. Selecting records to print 
would be another area that would benefit from multiple forms of 
access, as implemented with the use of a button or the return key. 
Many users tried to indicate which records they wanted printed by 
using the "Start printing from record #" line in the print dialog box. 
This could be changed to a more standard Mac word processing 
function of "print from record x to record y." The user needs a clear 
indication that their selection of records has been made before 
getting to the printer dialog box where several subjects used record 
numbers instead of page numbers to try to accomplish this task. 

The message "0 records formatted so far" when printing is 
about to begin caused consternation for several subjects. One 
impatient user lacking confidence cancelled printing at this stage, 
assuming she had done something wrong. This message could be 
eliminated, changed to read "formatting of records in process," or 
could appear with the first record being formatted as "records 
formatted so far: 1." For those who did not successfully select 
records, seeing the records formatted number count to successively 
higher numbers than they had intended was not a cause for concern 
so much as the appearance of "0 records formatted so far." 



The online index was underutilized and could have made the 
searching task easier for several subjects at all experience levels. 
This important feature seemed to be hidden from users who were 
familiar with the menubar as well as those who were familiar with 
the concept of an online index. One user who actually looked for the 
index stopped after looking under the first three menubar items. 
This could be made more prominent by the use of a button with text 
or an icon on the search screen. The selection of hyphenated terms 
would decrease searqh time, a major cause of dissatisfaction in this 
study. 

Because five subjects took the time to thoroughly read the 
introductory screen, I would suggest that this screen be used to offer 
searching tips and point out features of the software interface. If 
presented in an uncluttered, visually interesting way (such as with 
bulJcts), more users might read this and have a more pleasant 
searching experience. The presence of an online index could be 
touted here, as well as tips on accessing secondary menus under the 
help menu, using the context-sensitive help function, and avoiding 
searching the term "education." Experienced users may always 
bypass such information, but even a casual Mac user could benefit 
from pertinent searching tips. The inclination to read an 
introductory screen clearly existed here and could have been belter 
used to improve the product's efficiency. 

All ten subjects successfully completed a search and printed 
results from a CD-ROM which they had never searched before. That 
in itself is praise for any product. As students in a masters' program 
in library and information science, these subjects may or may not be 



typical users of MacSPIRS. but their experiences in this study 
presents valid design issues when a product is being developed or 
upgraded. 

Conclusions and Directions for Future Research 
By creating a descriptive taxonomy from a single community's 
use of a single product, this study hopes to be, as Shneiderman 
(1987) suggests, "a small and useful contribution to the emerging 
mosaic of human performance with interactive systems" (p. 79). 

The CD-ROM interface studied here employs all three of Piaget's 
stages as Kay (1990) urges designers to do. They were successful to 
varying degrees, however. 

The enactive stage, particularly the use of the mouse, 
presented a significant problem to only one of the ten subjects. Even 
this one subject progressed considerably in ease of handling the 
mouse through the one short search session conducted here. This 
differed from the findings of Ziegler and Fahnrich (Shaw, 1991) who 
found consistent problems with mouse manipulation for the novice 
user. 

The use of the iconic stage could be enhanced, examples being 
the lack of online index use even when subjects tried to find it and 
problems selecting records to print. A well-designed icon for the 
index, or as Herther (1986) suggests, integrating it into the search 
process, and a button with better wording for selecting records are 
possibilities. The GUI as implemented here was successful in that it 
removed the burden of remembering commands from tlijc user; 
however, users experienced problems remembering the location of 
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commands hidden under the menubar (such as print), and seldom 
used the Boolean operators listed as the buttons they were, but 
rather as reminders that they were available. 

The greatest problem within the symbolic stage was the 
confusion over how to return to the search window from displaying 
records, referred to as the navigational error. Eight of the ten 
subjects experienced this error. Clearly, the process did not 
correspond with the users' mental models of the system. Again, the 
addition of an icon or button would eliminate or reduce these errors. 
The consistent Mac interface appeared to aid Mac users, as Shaw 
(1991) has found, but caused some confusion for those unfamiliar 
with Mac standards. 

The implementation of Menu-Based Natural Language was not 
always helpful to the subjects here. As mentioned above. Boolean 
operators were within view of the user, which was a good reminder 
that they were available, but the efficient use of buttons here is 
questionable since it involves removing the hands from the keyboard 
to use the mouse to click for the insertion of a two or three letter 
word. It seems easier to just type the Boolean operators. Another 
example of confusing implementation is the Select Record button. A 
more explanatory label might be Select To Print. 

No choice of interface was offered for those with differing 
levels of experience, as suggested by Shneiderman (1987). This 
proved to be a problem for both the expert and the novice in this 
study. 

Shneiderman's (1987) eight underlying principles of good 
interface design are well-illustrated in this study: 
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i"?.^^' consistency. The implementation of the 
standard Mac mterface was a plus for Mac users. The 
consistent layout of the screens was received positively 
by most users. ^ ' 

2. Enable frequent users to use shortcuts. There was no 
provision for this, though none of the subjects here were 
frequent users. 

3. Offer informative feedback without confusing error 
messages. Error messages were often just beeps with no 
explanation. *^ 

4. Design dialogs to yield closure. The navigational error 
represents the greatest hindrance to the user's sense of 
accomplishment. 

5 Offer simple error handling. Error handling often used 
Mac terminology with which IBM users were not familiar 
such as "window" and "drag". An error message 
encouraging use of the online index would be appropriate 
instead of simply informing the user that the search was 
too complex. 

6. Permit easy reversal of action. The users could cancel 
searches and prints at will. One subject consistenUy 
cancelled searches because there were zero hits with the 
search less than 100% complete, which had no relation to 
the outcome of the search. 

7. Support internal locus of control. This is questionable 
considering the navigational error and the discomfort 
(even frustration) caused by the long resDonse times. 

8. Reduce short-term memory load. This would be 
further enhanced with the addition of print, index, and 
return to search buttons or icons. 

The methodology used in this study, though commonly 
employed for such research, still presents some problems. Some 
subjects were visibly nervous, had problems thinking aloud, or 
admitted that they would have done things differently had the 
environment not been so contrived. Some examples from the 
transcripts illustrate this clearly: 



It's just thinking that, urn, you know talking out loud is, 
uh, kind of confusing me. I was thinking if I had some' 
peace and quiet I could figure out what's going on more. 
I think I go too, I tried to do it the way I thought would 
be, the way I should do it, but I usually just go in and put 
any two terms in there, sec what I come up with. I think 
if I was going to do this I would have read the 
instructions before I started. If I were paying for it I 
would have planned the search ahead of time. I also 
would have spent some time looking through the 
hardcopy manuals. 

This may indicate a link betv -n a subject's degree of introversion 
or extroversion and the implementarion of the think-aloud protocol. 
This could be an area for future research. 

Perhaps only covert observation would reveal if subjects would 
really have consulted hardcopy reference materials or more carefully 
planned their strategy before searching. I have no doubt that some 
of the subjects in this study would have quit in frustration had I not 
been observing them. It is impossible to guess, however, if they 
would have sought help from reference material or a person before 
abandoning their efforts. 

Other suggestions for further research are following: 

• study a similar or dissimilar community to compare 
results, 

use a counterbalanced design with subjects 
performing a search on both IBM and Mac platforms, 

• narrow the subjects to only those with a Mac or 
IBM preference or to those with equar searching skills, 

gain access to the code, change one thing and 
measure the effect, such as integrating the online index 
into the search process, 

return to Shneiderman's (1987) list and design 
studies to measure the time needed to learn the system, 
the speed of performance, or the retention over time. 
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APPENDIX A 
CODED TAPE TRANSCRIPTS 



Key: 

Normal type is tape transcription. 
(Interviewer's comments in brackets.] 
Interviewer's observations in italics. 
BUTTONS ON SCREEN IN ALL CAPS. 
"What subject typed in quotation marks." 

Codes, in alphabetical order: 

E error 

Ex searcher explores system functions 

H use of help screen 

I reading of introductory screen 

Idx use of online index 

L logic error 

M problem with mouse 

Mac Mac instruction offered by interviewer 

NE navigational error 

N+ intuitive navigation 

Q use of ? icon, for context-sensitive help 

use of quick reference card 

RT response time issue 

S+ system interface pluss 

S- system interface minus 

TH use of ERIC thesaurus 

UM use of user manual 

? searcher doesn't understand 

Session 1 

This is a novice searcher with no Mac experience. She has 
experience on an IBM PC at home, though. She asks what 
Boolean is and seems familiar with the concept. 
I She takes time to read the introductory screen before 

beginning anything. She makes no notes on her search. She 
has no previous searching experience. 

OK. All right. All right, I'm starting the, urn, search now. And 
I'm going to um start by typing in, um, "elementary education". 
Sf And I'm going to hit ENTER. RETURN, actually on the Mac. And 
on the screen, now I have, um, appearing records in 
elementary and records in education and records in elementary 
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education. She is obviously going to get a lot of hits on this It 
doesn't occur to her to BREAK SEARCH 
E The system is still, urn. searching. I'm going to ask for. ask for 
the records so far in. urn, well I'm going to try #3 records in 
elementary education, although it stys therc^c no^^ct^^^^ 
there. Nothing happened. fccoras 

^ fZri'T ni try pressing #1. elementary. Nothing happened 
there either. Maybe I need to wait until the search is 
completed. 

E I may press the. urn. question mark. She hits the question 

mark on the keyboard, she does not click the icon ?. Although 

hTre'eithr 'm"'' ^ '^^P'"^' ""'^^ "^^^^^ ^^^^^ 

tnere either. Hm. Moving the mouse over to the help area and 

7h?ZsT '''' ^^^^ -^-2 

? Something has appeared on the screen called BREAK SEARCH 
but Im not sure what that means because it's still searching' 
ok' riL ' f '"f "^''^ ^"^^"^^^^ Ok. lefs^see. 

^e thinkTngT Sk' " ''^"^ ^^"'^ "^^ ^''^ 

CR I'm looking at the. urn. SilverPlatter Quick Reference Guide, to 
see If It can give me any helps here on how to move from 
whats happening now on the screen which is basically the 
system is still searching, 66% complete at this point, urn but I 
rea ly need to move into something more specific, so Fm 
ookmg at this guide to see if that's going to give me any help. 
I see a searching free text, um. information. That tells me to 
press F2 to display the find prompt. I'm not sure if I can do 
that from where I am at this point. Well, let's see, I'll try it 

S^keylXl'don't^'^^^^^^ '''' 
Mac [Let me inten-upt you here. If you'll use the mouse, and move 
this down to BREAK SEARCH, and then click it.] OK. T^lenTat 
will stop? Ah, OK. I moved the mouse down to BREAK SEARCH 
and now we have something that says SHOW RECORDS FOUND 
So I think we'll try that, and we'll see what... Ah. OK, now this 
IS going to show me everything it found, which is seieral 
thousand files at this point, and I don't think I want to look at 
eveiything. so I need to get something a little more specific. At 
the bottom of the screen we're seeing NEXT RECORD PRFVinriQ 

or SELECT RECORD. She is careful to read the screens. Um, let 
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The urn the guide for the uh SilverPlattcr Quick Reference 

Guide rm not sure that this was designed for use with the 

Macintosh computer. Perhaps it was, but I'm not sure that it, 

urn, would appear that it was not. OK, let's sec 

I want something more specific, so let me try SELECT FIELDS 

and see what that does for me. Uh. nothing changed on the 

screen, I still have this first file on, uh. a p'rogress'^p^rt o„ a 

funded program on the educational, instructional program for 

tiie TV District of Columbia public schools, which hasVo^ing to 

do at all with the topic that Fm. uh, interested in. 

She has trouble putting the mouse inside the button. She has 

about It? She puts the mouse pointer at the top edge of the 
button and it obviously doesn't work, but she fails to interpret 
a lack of response as an incorrect mouse click. I'm coine to trv 
urn, something called SEARCH FROM RECORDS 
The screen says click on any word or drag across any phrase 
that you want to use in a search. Hm. Click on any word or 
drag across any phrase that you want to use in a search I'm 
not sure what that, urn, what that means 

I'm not sure if I can type something in, the word or phrase that 
I want. Ill try It and see what happens. No, nothing is 
happening. Hm. * 

Well there's a box that says OK at the bottom of this little box 
1 11 click on that and see what happens. I just, it just came back 
to the same, um. the same item that was on the screen before 
Uts see what happens, SEARCH FROM RECORDS None that ' 
^iml^T'^^JMf ' had before, um. OK. let's get to SHOW 
SUBSET. Well see what that does. Ah, well, FU cancel that 
because what is shown on the screen is show subset, selected 
records, all retneved records, or skip to record number, and 
smce I don t know any record numbers I want to skip to. and I 
don t want to show all retrieved records, I don't know, maybe 
selected records, maybe that would be a thing to try, let's see 
I m sure there s an easier way to do this. Well, I pushed on 
selected records, but nothing happened. Selected records is not 
showing as an option since she has not selected any records. 
The systems beeps, but no error message appears. Since it's 
blinking on skip to record numbe- I assume that's what they 
wan^ and since I don't know a record number Fd better get out 



uh is a question, maybe I could, 

uh, lets see if, let's see if that gives me any information there if 



I click on question. No, same, except now I have a question 
mark with my mouse. This is where field sensitive help would 
come up, but there is little indication that that is what this is 
supposed to do. 

Nf OK, now I'm back to the arrow. I just pressed the clicker on the 
mouse again and got rid of the question mark. Fm going to 
cancel this. OK, now Fm back to where I was before. 

M Very hesitant with mouse movement. Uh, Fm going to check 
one of these other reference things I have now. 

UM I have a, uh, a binder entitled, ah, the MacSPIRS User Manual. 
Now that would have been something to start with. Looking at 
the getting started, MacSPIRS is already running, let's see what, 

Mac [As a Macintosh novice, let me point you to one other source. If 
you move the mouse up here to this menubar, you can click on 
that, and then go down to select one of these topics, OK? Help. 
OK? And those arrows mean there is more over here.] All 
right. [Then if you lift up on the mouse, you'll be where you 
stopped. See that? OK?] All right, thank you very much. OK, I 
think I will go to the, um, help area at the top of the screen. 

M Oops, OK, I need to hold the little clicker down apparently in 
order to have those items show, 

H We have Getting Help, Guide to ERIC, Finding records. Showing 
records, IMnting records. Saving records, etc. Uh, and Fm going 
to go down to the Guide to ERIC. OK. Hm. 

M Apparently you need to keep the clicker down, all right, all 
rights now I'm going to move over to the Introduction in the 
Guide to ERIC. 

M Trouble selecting from a secondary menu. Well, I was. All 
rights now I'm moving the mouse again into the Guide to ERIC 
and there seems to be a narrow band that one needs to move 
the mouse in in order to get the assistance. And that's a littie 
tricky. 

E And for some reason, hm, well now Fm back to my search 
history and the original search request. By mistake, she gets 
back to the search screen she's been trying to locate, but she 
continues to try to get into help. Fm still up here trying to get 
back to the help. OK, all right, now I'm moving the mouse 
down to the Guide to ERIC. 

M All right, and I'm going to try to move the mouse across in this 
narrow band, but Fm going, it's going to take sorne doing, and 
perhaps you don't really need to do this, but I don't know. 

M All right, I'm in the Introduction now in the Guide to ERIC, but 
I don*t know how to get in there because I have held the 
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clicker down to get over there and when I release, hm. ok, now 
Its all nght. Wc have the ERIC introduction on the screen now. 
And Its telling me what it is. And I assume, let's see, to sec 
more of the screen, and I'm not sure if I press, no, there are no 
arrow keys, urn, OK, do I have to get over, do I have to get the 
mouse over here to the, uh, there. 
M Painfully slow with the mouse. Now let's sec if the ok all 
right. ' • 

Nf I moved the mouse over' to the far right of the screen to the 
arrow that goes up and the arrow that goes down, and that's 
how Im moving the, uh, moving the page. She easily figures 
""u ^^J^'^"^""^ window. All right. And it's just telling me 
what ERIC IS. And it's telling me what's contained in ERIC And 
rm just clicking because it's giving me lots -f information that 
Im not interested in. Here we get to the diabase structure, 
hm. The last thing appearing here is that each item or 
document in ERIC is called a record, all records are divided into 
fields, see guide of fields for a list of fields in ERIC. Apparently 
this is all there is in the introduction. OK. So now to get out of 
this, I think I'm going to look at the guide again 
Mac [I'll help you out here.] Oh, OK. [If you click this little box in 
XM ^^^^^ window, it will close the window.] 

M All nght, but I have to get the arrow up there, right*? 
Mac [When you get to the end of the pad? Pick that up and move it 

back down.] Oh. Oh, OK, all right. 
M All right, I'm leaving the ERIC introduction. At least I think I 
am. [The arrow has to be really inside the box.] Hand- eye 
coordination isn't good Wonder if she's used a computer or 
video game at all? All right, so now I'm back again to the 
initial search history, so if I want to go over and get a little 
more help. 

H I'm going back over to the help area and this time I'll need to 
choose a different, oh, I think Finding records this time is what 
I m going to go to. And I'm moving the mouse down, and I'm 
in finding records, and I'm going to go over to starting I'm 
moving the mouse to the right, and I'm going to start a search. 
UK. 

L Interesting that for help she picks starting a search as opposed 
to narrowing a search which is really where she is in the 
process. Did she not read the entire list before choosing? AU 
right, now we have information on starting a search, telling 
that the find records box appears at the foot of the screen 
when you first start up MacSPIRS, and for the simplest of 
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searches, type a word, click the START SEARCH button and 
you 11 see the results in the find records box. OK, let's, my 
initial word was far too broad. 



I am having difficulty learning to use this mouse properly OK 
Her entire body leans as she moves the mouse. All right.* now* 
Im stiU continuing with the explanation on the screen of 
starung a search. It's telling me that if my search request 
contained more than one word, the system would repeat 
intermediate results for each word, treating each word as an 
individual search request before searching for the combination, 
so thats why I got records for elementary and education 
before records appearing in elementary education. OK Um 
and now it tells me that we could search with a term or a 
phrase, a hyphenated phrase, etc. What I'm looking for 
specifically is how to get, um, once you've done a search in a 
particular, um, in a particular term or phrase, how to then 
narrow that down, and this information on starting a search is 
not going to tell me that, so I'm going to get out of this now by 
again moving the mouse up to this little box. We'll see if. OK 
I'm going to have to go back up to the help. All right, I'm 
clicking on help again. Apparently my guide and finding 
records is not specific enough, so perhaps Showing records is 
well, let me look at Finding records again. Ah. there we go 
Narrowing a search. 

That's exactly what I need if I can. get over there. Oh dear. OK 
Now hopefully I'm on narrowing a search. Nothing is showing 
on the screen, ah, here we go. narrowing a search. 
RT She gets very nervous here when the system response is 

slower than expected. All right. Right here, there are several 
ways to narrow a search, browse through the index, combine 
words or phrases, using and not or with or you can search in a 
particular field, operators are discussed below. 
Nf OK, let me. all right, moving the screen down, now, most 
databases have specially indexed limit fields, example 
language, publication year, that you can use with other search 
terms. Use the and operator to retrieve all records containing 
two or more specified terms anywhere in the record, for 
example drug and abuse would retrieve all records containinc 
these two words. Um. all right. I see. use the with operator to 
retneve words, records in which two or more terms appear in 
the same field. All right, so my search is on nutrition and 
exercise. Um. so then I could search in both nutrition and 
exercise and nutrition with exercise. OK. And by using a near 
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term I could search for. urn. how close the term should be for 
example. All right, and the not. using the not operator would 
retrieve, urn, one or the other. Finding a term or terms in a 
particular field. All right, now it's telling me how to find a 

l^frJ^"^^^. ^ ^"^^ Searching 

imit fie ds. All nght. I'm on the section entitled searching 
limit fields, and this is telling me I can limit my search 
perhaps the equal sign or in. Oh. all right, if I wanted io search 
for records m a particular year, or earlier or later than a 
particular year, or equal to. or specifically in a particular year. 
I don t need that, so I'm going to move on. Uh. I could limit it 
further by using the. uh. operators and. or. with, along with the 
urn. uh greater than, less than, equal sign with the year, so that 
would be another way of of narrowing it more specifically. 
OK so this IS telhng me how to, urn, the terminology to use. I'm 
sun unsure how I move from where I am now. into something 
narrower. Lets see if there are any other, any other, um. 
guidelines here that are going to help me. She is so diligent 
about reading through help for narrowing search, but still 
unsure how to do it^ She needs more specific information about 
what to do next. OK. I'm going back up to the help again, 
although we have down at the bottom of the screen now. enter 
search and combine terms using and or not with or near so 
maybe I'll just go down there and try. 

All right. I'm going to try pressing the term AND and see if I 
get something. Hm, no. I didn't get anything. / think she 
clicked on the word "automatically" when she was displaying 
records originally, and now it is in the search box. When she 
presses AND. she now has "automatically and" in the box but 
she hasn't noticed yet. 

All right. I'm going back up to the help. Sigh. I don't think 
there is anything in the finding records that is really going to 
help me. or maybe I'll try showing records this time. AH right 
let s see. showing records, my options here are using show * 
records, selecting fields, selecting records, displaying field 
labels. I don't think that's going to help. I think there's 
something. OK, let's. I think maybe what I want to do is clear 
this record, so maybe let me try that. OK. This is going to give 
me information now on clearing a search, it erases search 
request from the search history window. To erase unwanted 
searches, select clear searches from the search menu, type the 
number of the last search request you want to erase and press 
the OK button. Um. hm. I don't know if this is really going to be 
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what I want or not. I can see, um, from this experience that 
attending the, um, uh, ERIC introduction course at the education 
library at OSU would probably be a fine thing for me to do, um, 
and also, I think that without that it might be possible for 
someone to get a little frustrated. [Are you frustrated?] No' 
really, Fm just, um, you know, what I really want to do is 
somehow just perhaps going to the indexes themselves, the 
paper indexes, I would have saved time, at least at this point, 
but, OK, Fm going to clear this. Now let's see, if I want to see 
the, Fm still real unsure how to narrow the search down, 
because I don't want to look at all 74,000 records that they 
found in elementary education. 

EX If I go to search here, on along the top of the screen, let me see 
what happens. Find records, browse index, list field names. 
Much more comfortable at this point exploring menu options. 
All right, good. I ±ink that may be what I want to do. OK. 
Well, Fm not sure that is. I don't think that's what I want. 
What I have here is a list of field names, and I don't think 
that's... OK, let's see. We're back here to search, now OK, find 
records, is this going to, probably that's what Fm doing now. 

Idx Browse index, oh, here. 

E When index comes up blank, she easily sees the entry line for 
word or root, but types in her search terms as opposed to an 
index entry, OK, enter the word or root that I want, OK, let's 
see. Fm typing in "nutrition and exercise". Fm going to press 
enter, ah, and it's telling me that only single words and 
hyphenated phrases can be looked up in the index, and I 
thought that you could use both, but perhaps not. Um, hm. OK. 
Well, let's see. 

E Well, I was trying to get rid of the 'and exercise,' and again, I 
guess because of my inexperience with this Macintosh I've 
succeeded in erasing nutrition and the 'a' in and instead of 
what I wanted to do. The clear button is not helping me, at all. 
She uses the Clear key first, then backspace to delete. The 
clear key inserts a square. And there is no delete button on 
this Macintosh, so Fm not sure how you get rid of mistakes. 
There is a delete button on my SE keyboard at home. Do you 
want to tell me? 

Mac [Uh, yeah. You can either drag, ok, drag across there, and hit 
backspace, will delete it or if you have something in there and 
you're here* and you can move these arrow keys down at the 
bottom, then use your backspace key, either one.] OK, thank 
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you. All right, I'm going to type in, the word just "nutrition", 
and we'll see what happens. 
Sf Press the return button, OK. Now I have lots of things showing 
up, um, in a box under the index, urn, nutrition alone, urn, 
nutntion and gerontology, nutrition and school, nutrition and 
education. Uh, let me see what, what we find what nutrition- 
education. 

E Let me sec what I'm clicking on nutrition-education-and, I 
don't know if there's something that comes after that or not. 
Hm. The bottom of the screen, we see the last part of the word 
automatically and nutrition. [Let me get you out of this 
because I know what you did, but you didn't notice it.] OK [I'll 
just get rid of that for you.] OK, now I see at the bottom, 
nutrition education and training program, that's not really what 
I want, um. I'll just try nutrition and education. 
E Even though what is selected from the index isn't what she 
wants, she doesn't hit ERASE before selecting another. Hm. 
Apparently I needed to erase what was there before, before I 
tried a different term. Hm. OK. OK, I'm back to nutrition 
education and training program, I don't think that's what I 
want. I'm tempted to go down here to the bottom and press 
ERASE and see what happens. Ah, OK. That, um, erased the 
particular term that I was clicking on, and I'm going to try now 
just simply nutrition and education. All right, since there are 
only three records in nutrition and education, I'm going to 
show the records now and see. what there is, and see if it uh 
hm. ' ' 

E After figuring out the ERASE button, she doesn't START 

SEARCH. She still hasn't figured out the button concept, I think. 
But all I got was the initial record that I wanted in elementary 
education, I did not get the, uh, particular record in nutrition 
and education. 

E I'm going to try here clicking on SHOW SUBSET and see if that 

will give it to me. But I don't think so, no. Goodness. Sigh. 
H I'm back up now to help again, because... No, that's not what I 
want. Search. Show Searches from menubar finally gives the 
screen she wants. And I want to find records. No, I guess I 
don't, oh, in nutrition and education, yes, which showed up at 
the bottom. Oh, I know, I need to start the seaich in that area, 
OK. At the bottom of the screen, I'm going to press on the box* 
marked START SEARCH. All right. 
E So it found three in nutrition and education, and now I'm going 
to click on nutrition and education and see what happens. And 
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nothing happened. Actually, it moves the terms into the search 
box, but she doesn't notice. And if I go down to, SHOW THE 
RECORDS FOUND... All right, right, now I have, OK, I have a 
record now in nutrition and education, the first record, which is 
an article on the effectiveness of radio nutrition education. 
Ni- I don't think that's going to help me very much, so I'll go to the 
next record. More comfortable with mouse at this point, easier 
to select buttons. All right. Ah, this one, an inventory for 
assessing food behavior of elementary school children. It's 
possible that that would, um. Yeah, it's possible that that 
article might. Now, let's see, I might, I'm going to go on and 
look at the third um, the third article that the system has 
found. Ah, this one on the development of a nutrition inservice 
course for teachers of young children, now that could be 
helpful, as well. All right, so I think that two and three would 
be um, might be helpful sources, so let's see I want to, since I'm 
supposed to print five of them [if you find five], yeah. I will 
print those two at least. Now I need to figure out how to do 
that. Um. OK. 

H I'm in the help area, and I'm moving the mouse down to 

printing records. OK. I'm going to click on selecting records, for 
printing, I'm hoping that, OK. I can print it by clicking on any 
record number, and when you click on the record number it's 
displayed in bold. All right, when you've made a record 
selection, OK. The print records dialog box offers the option of 
printing all retrieved records or just selected, the dialog box 
will offer the option of specifying a record number from which 
you want the printing to start, ok, that's what I want to do. All 
right, so I need to first get out of this help box. 

Nf Now, I'm going to move my screen, I am now on the, um, third, 
um, record, that was found and I'm going to move to the 
previous record, ok, now I'm moving the mouse up to the 
record number 2 of 3 and I'm going to click, ok, and that's 
appearing in bold now, oh dear, and I don't think I've found 
now how to print. 

Nf If I click this again, I believe that says I don't want to print it. 
Let's see. Yep. 

H Better go up and read the help again. Oh, OK. Ah, OK, then I 
need to go to the show subset dialog box, after I click on the 
record I want to print and that's what is going to, all right. 

E Still a bit difficult to select secondary menus in help. She got to 
that help screen by mistake, the mouse slipped when trying to 
select Printing in the help secondary menu. All right, I'm back 
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on the screen again, it's my second reference, I am now clicking 
on that reference, and it's appearing in bold, and I'm going 
down to the show subset box, and I want to print selected 
records. So that's what I'm going to click on. Ah, hm. 
She discovers radio button by clicking and deselecting it. 
Figures out the toggle idea easily and re-clicks to select. All 
right. I'll go with OK, and I'll see what happens. I don't know 
if that's printing now or not? [No, it's not.] No, OK. All right, 
well, I did something wrong then. Sigh. All right, I'm going 
back to the show subset box. Hm. [I'd encourage you at this 
point, to go back to your help and try that one more time.] All 
right. She is, coughing and seems bothered. [Would you like to 
take a break and get some water?] Oh, no, I'm fine. 
OK, I'm back in the help. And I'm going back to the printing. 
And I'm going to try this, let's see, using print records, let's see 
if that will tell me what I need to do. All right. It's telling me 
first that I need to be connected to a printer, and I assume that 
I am. If I want to print records found by a previous search, I 
must make sure that I make that search the most recent by 
repeating it, the easiest way is to click on the number of the 
search you waxii to repeat, OK. Then select print records. Hm. 
All right, well, if I saw print records on here somewhere that 
would be helpful, but I don't. OK, apparently, this isn't going to 
answer my question. [Try scrolling back up to the beginning of 
that.] OK. Moving back to the beginning of the, uh, this 
explanation. Ah, OK. I can print the record by selecting the 
print records from the file menu, all right, I see then that's 
where I need to go is to the file menu. So that's what I'm 
doing, I'm going now to the file menu. And, I want to print the 
records. All right. Ah, OK, I want to start printing from record 
number 2, I also want to print record #3, I don't know, let's 
see. I want to print selected records. Hm, well that doesn't do 
anything, OK, well the screen says start printing from record 
#2, and that's what I want to do, so I'll print that one. 
She has a problem getting the arrow in the radio box. And, I 
want one copy. And I'm going to say OK. 
Hm, OK. I, did it print tiiat? The printer is not close in 
proximity here, but the message should have made it clear that 
it was printing. I think it printed that particular reference. 
Now, I guess I'll go to the next record and print... 
[Beep.] Hm. For some reason it's not going to the next record. 
[Beep.] And I don't know why. Let's sec if it will go to the 
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previous record. [Beep.] The system beeps, but there is 
absolutely no indication of what the problem might be. No. 
Sort of stuck here in reference 2. Hm. I'm not sure why I can't 
get out of this. 

I'll go back up to my help box again and see. Something called 
restarting the session. Maybe there's something I need to do in 
that. Use the restart session in the file menu at the beginning 
or end of a work session. Um, I don't know. I'll go back up to 
the file menu, and see what's in there. Maybe I was supposed 
to do something after I printed that last, that last reference. 
We have save records, options for save, printing records, I do 
want to print, I want to print #3, but I don't know if I can get 
to it from this point or not, because I have not clicked on 
record #3 yet. OK, I'm in the print records box, and I don't 
want to start printing from record 2 which is what it says, um, 
I guess ru cancel this. I still want to try to get to record #3, 
I'm going down to the bottom of the screen where it says next 
record, and I'm clicking again [Beep.] and nothing's 
happening, it's just beeping. Let's see what happens if I go to 
select records. 

Select records by clicking on the record #, oh, I know what the 
problem is, I did not, I need to go back to record #2 and click 
again because in the bold, and get back to the regular type 
instead of the bold. OK, good, now I think I can go to next 
record. [Beep.] No, hm. Can I go back to the previous record? 
[Beep.] No, let's see I didn't read enough in select records. I'm 
reading the text on the select records box. Sigh. It's not 
necessary to click the select records button before selecting a 
record, OK, but click again to deselect a record, which I did, I 
thought. 

I'm going to go back up to the help box and see if it, see if it 
tells me anything under showing records. Um, I'm going to see 
something about selecting records here under the help. OK, 
when I've made a record selection, the show subset dialog box 
offers the option of displaying just the selected records or all 
retrieved. Oh, show subset, well let's see if I can, I don't think 
I can get to it from there, but I'll see. I'm getting out of show 
records and I'm going to go down to show subset, and I don't 
think that's going to help me at all. OK, now I can skip to 
record #3, let's see if I can really do that. I typed #3, entered, 
ah, and here we go, wonderful, wonderful. Now I want to print 
this thing. #3. So I'm going to click on 3 of 3, that puts it in 
bold. Now I'm going up to file. And I want to print this. I'm 
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clicking on print records, and I want to start printing from 
record #3, so I'm pressing print. I want one copy. And Tm 
clicking OK to print. OK. Sigh. So we've printed two records 
now. [Would you like to stop now?] It's up to you. [You've 
done a fine job.] Well, I'm not sure about that. Yeah, why don't 
we stop at this point. She relied heavily on help, but the topics 
weren't as specific as she needed. It was frustrating that there 
were no messages to assist her when she needed it. 
QUESTIONNAIRE COMMEl>rrS: Lack of familiarty with a Macintosh 
computer and the mouse compounded my problems with the 
program. I think it would have been more helpful to me had I read 
the instruction manual before beginning the search. I assumed the 
systeni was similar to an automated card catalog where after 
beginning a broad search you can easily narrow it. I found 
narrowing this search to be incredibly complex, but again reading the 
manual first might have given me more specific direction. 

Session 2 

Some Mac experience, no searching experience. She is checking 
over the reference guide and making some notes for search 
strategy. 

I She is reading the introductory screen and looking through the 
manuals. 

Well I'm just going to kind of experiment with it without 
looking at anything first. And then see what happens when I 
do it. First I'm just going to type in "obesity and children". 
Sf Note that "and" is typed, not selected from the Boolean buttons. 
68 records found on that. So we'll see SHOW RECORDS FOUNT. 
Let's go to the next one and see if it has any summaries or 
anything like that. See, here would be one, teachers' 
perceptions of childhood obesity. That would have to deal with 
teaching young children. Oh, here, I didn't even see this down 
here about the childhood needs. Education and the elementary 
school student. The journal is health education. Go to the next 
one. Another one you would probably want to look at. Down 
here. It deals with elementary and secondary education. So 
that would just be another one. I'm not going to go through all 
of these because there are 68 of them. Just maybe 10 of them. 
This one, too. Is there a crisis in youth fitness, that would have 
to deal with it. Next one. Next one. 

RT She is impatient when waiting to go to next record. 

E Extra click causing skipped record. Right here, childhood 

obesity, concern for the physical educator. That would have to 
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deal with it. Very proficient with mouse. Well, I was going to 
do about ten, but I see I'm on about 25, so I'm going to go, I 
don't know, see what this is. 

E Continues to click when clock icon shows working - many 
records skipped. 

EX SEARCH FROM RECORDS. SELECT FIELDS. See what that is. 

E I did something wrong. [System bombs.] System bombs 

because of repeated mouse clicks before the system was ready 
to accept them. I don't know if she figured this out. She is 
confident, but nervous. I put the CD back in, then I'll just 
restart it. 

Mac [If you'll double click on MacSPIRS.J She can restart without 
problems, again confident. OK. This time when I do my search 
I'm just going to use two different words. Urn, probably 
nutrition, or maybe just obesity again, and education, this time. 
[I want to encourage you to think aloud. Let me know what 
you're looking for or why you might be...] 
Ql Refers to reference guide for help with Boolean operators. OK. 
I was looking to see if you could use three words instead of just 
two, and then I'm looking to see now the differences in the 
terms and or, because the first time I did obesity and children, 
um, now I'm looking to see what maybe or or with is. OK. I'm 
just going to type in something different now. Nutrition with 
exercise, no, nutrition with obesity, or "obesity with education". 
TH Checks thesaurus briefly, just to see what it looks like. 
UM Peruses getting started manual while "obesity with education" 
search continues. Now I'm going to get adults in there too, but 
at least I can see how many I'm going to get. What I'm trying 
to do, I guess I should probably look and see how, once I get 
this, how to get children in there also. Found 55 of them. Hm. 
Now I'll see if there's any way I could take obesity with 
education and add children in there and search that also. 
UM Flips through Getting Started to Reference manual. There is no 
use of index or table of contents. Here we go. Probably just 
type in #3 with children. So there are 9 records. So that would 
probably be a good indication for finding out what you want to 
know. I'll show the records found, see if it is dealing with what 
you want to know. This one would relate to it. No use of scroll 
field to see if abstracts are available. Go to the next one. 'Title 
childhood obesity a concern for the physical educator. I might 
have had that one before, also. I don't remember. Title 
childhood obesity. Right here, the title of this one, relationship 
between nutrition and learning, a school employee's guide to 
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information and action. Childhood obesity, implications for 
physical education. I want to go back where I can search. 
[Prompt you to think aloud. 1 

NE At point where she wants to go back to search screen, she looks 
'under' buttons to see what is there. No use of menubar yet. 

BC OK. I'm just reading this now. Just trying to see what some of 
these things are now. I don't know if I looked at SEARCH FROM 
RECORDS. I want to get back, but I don't know how to do it, 
unless I just keep hitting PREVIOUS RECORD, 9 times. There 
has to be an easier way to do that. See if it goes back to 
previous record. No. Looking at the menu up above to sec 
what everything is. She browses menubar items -- bypasses 
quickly Search and Show to Help, but finds Narrowing a Search 
unhelpful. 

H Looking under help. Going under finding records to narrowmg 
the search. 

M Oop. I need to go up one. She has trouble with secondary 

menu, but fixes it easily. See what it says. That's not helping 
me do what I wanted to do which is to go back to the 
beginning. It's just telling me kind of what I knew already. 
OK. I do want to get back. I still have to figure that out. 
UM Looking in the book now to see if there's anything, but I'm kind 
of glancing through it, I'm not going to read it. It says restart 
session, but I'm not sure I want to do that, because it might 
restart the whole thing, and not restart me to where I have 
nutrition, or obesity with education. Might. 
Idx Hm. I'm going to see what browse index is, and here I have it 
back, but that's not what I want either. [I'm going to encourage 
you to think aloud. Tell me what you're doing, why you're 
doing it.] Browse index, first confuses word or root entry with 
search screen. OK. Um, I'm going to enter the word nutrition in 
there to see what it does. I never put that in there. I don't 
really think, I think it's going to confuse me more than 
anything else. Oop, went too far. 
Nf In index, uses right scroll arrow to view terms in entirety. 

Nutrition at school, nutrition education, things like that might 
help. Well, I'm out. I don't see how it's going to get me back to 
where I want to go when I get out of there, out of the index. 
H I'Hi going to look under help again. Going to Guide to ERIC. To 

examples, see what that does. 
Ql Up here in the examples, it is giving examples, and I don't 

know what some of the abbreviations are, which they are here 
in the reference, which I forgot about this, actually. I'm 
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leaving that little help thing. Sweeping movements of mouse 
occasionally. 

Then, under help, restarting a session. See, it says, clears the 
search history, so I don't want to do that. And that's done, so I 
know that's not what I want to do. 

Search, see what options for find is, display intermediate 
results. Gonna cancel that. List Held names, see what that is. 
Oh, I saw that ahready. Cancel that. Find records. See what 
that is. Here we go. Something else. Showing what I searched. 
And I can search, and add to it, which is what I was looking for. 
So I want to see what else I want to add. Right now I have 
obesity with education with children. Um, if I just maybe want 
to try to add to that, do a #5 and nutrition. 5 records. 
Types #5 -- no use of clicking in history window. SHOW 
RECORDS FOUND. Uh, saw this one already, can we help these 
children. Saw that. Go to next record. Saw that one. I think 
these are probably all ones I've seen, I guess. Yep, saw that 
one. And the last one, saw that. [Keep thinking aloud. What 
are you thinking.] OK. [What do you want to do next?] 
Reminder to print records, she seemed at a loss of what to do 
next. I wonder how well she read the instructions. I don't 
know, to be honest. 

I guess I'm really kind of playing with it to see what 
everything does, since I've never used it before. [When you 
get answers that you think answer the question, print off 5 or 
up to 5, don't print any more than 5.] Oh, OK. [Print out up to 5 
of them, what you feel like are good answers.] Well, I think 
I'm going to print these 5, because it's, I feel at least it's dealing 
with things you want to know about. The obesity, the 
nutrition, the young children, and the education. So I'm going 
to go to file, and select print records. Select fields, selected 
records, all records, include search history, no. Click on short 
names to select fields. All fields, OK. Seems ^he clicked on 
select fields just wondering what it was, then felt obligated to 
change something, and selecting them all was the easiest thing 
to change. 

I'm going to hit print. But no records were formatted so far. So 
I did something wrong there. No, maybe not. When no records 
formatted yet, she starts to BREAK PRINT, but she pauses long 
enough for the system to continue. One record, here we go. 
Now if it hits more than 5 records formatted then I probably 
printed something I didn't want to. As long as it does 5, there 
are 5 that dealt with all areas concerned. It did 5, so I'm 
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assuming it is the 5 that were up here that dealt with all of 
them. [OK*] Can I see if that was right? [You can turn off the 
mic now if you're ready to stop.] OK. 

Printed 5 of 5 so didn't have to discover how to select records. 
QUESTIONNAIRE COMMENTS: Even though I knew this wasn't a test, 
I still felt some pressure, nervousness. I think if I was more relaxed, 
I would have played with the system mor. and perhaps used the 
reference materials supplied. Overall the exercise gave me a good 
idea of what ERIC does, some basic instructions on using it. 

Session 3 

Subject has Mac experience with word processing. 
I Moves mouse as he reads introductory screen as you would 
follow print with your / \ger. 

OK. Well, I'm going to start reading these instructions so it'll 
take a minute. Uh, I'm going to search under the subject of 
obesity, I guess, um, I'll try obesity in children**. 
E Some problem with backspace! delete. He finally figures it out. 
[Beep.] Um. 

? I'm just going to search the topic obesity because, um, I don't 
understand what they're saying to do now. 

L Use of "in" ccused system to expect a field name next, not 
another search word. 

? Error message displayed, but he didn't understand it, so he 
changed to a simpler, one-term search. [Keep talking, tell me 
what you're thinking, what you're doing.] OK. Um, let's see. 
Um. I, I'm looking for articles on, on obesity in childhood, and I 
guess each one of these lines, the first two lines here I don't 
understand what they're about, but, I guess the third line is, 
uh, the citation for a magazine called Child Health USA, which 
sounds like it is going to have something that I want in it. AN 
and CHN fields are #7 and 2. The next article, um, maternal 
and child health services sounds like it might have something I 
want in it. The next article I don't understand the code though, 
you know, the letters. The next citation, I don't understand 
what that means, is either, um, let's see, maybe I'll read 
through these, I'll read through these search things at the 
bottom and see if I can make any sense out of it. Refers to 
each line as a citation. These are all fields within the first 
citation. Um, I'm going to click one of these articles and see if 
that's what brings L up, 

E Um, I'm going to click show records found and see if that will 
print the article out. I guess not. 
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? Um, urn, Vm not sure whether the machine has articles in it or 
whether it*s just, um, or whether this is just a list of articles 
that I have to go somewhere else to find. Um, let*s sec. 
Something tells me I should be able to bring up the whole 
article, read the article, but maybe not. Maybe this is just a list 
of articles. 

Ex Fm just going to, um, click all these various subjects at the 

bottom -of the screen to see what they do, to find out what they 
do. It looks like next record, um, oh, it looks like there's, um, 
maybe 228 pages of records on this subject, and Fve just 
clicked the third one, the fourth one. Hm. I wonder what 
select fields brings up. He reads the list of field names, but Vm 
not sure he connects it with fields in each record. Um. I'm 
going to cancel that, it doesn't, and try the next, Tm going to 
click search from records. And I'm going to click select records. 
Fm going to go back and read these instructions, and maybe I 
can figure out whether or not Fm just using, you know, a 
program that lists different articles or whether I can actually 
bring up the articles. Fm going to go back to the first page of, 
um, articles, of records. 

RT [Beep, beep, beep, beep.] No problem with mouse movement, 
but impaiient and clicks several times while clock shows still 
working. Now Fm going, I want to see this record, see if I can 
get the machine to print out this article from Child Health USA. 
And, um, I don't think it can. There doesn't seem to be any, 
um, I press select records and see what that does. 

Ql Hm, that didn't, Fm going to look in this, um, in this reference 
guide, and see if I can, um, find out whether this, um, search is 
just a list of articles or whether it actually prints up an article. 
[Tell me what you're finding. Is something there helping?] 
Um, let's see. There doesn't seem to be anything to directly 
answer my question. Um, whether or not I can actually bring 
up the article or whether I'm just dealing with something that, 
with a program that just lists articles. Um, I think Fm just 
going to write down, um, the first five articles that come up 
about children and obesity. Um, hm, OK. Oh, OK, the first thing 
that caught my eye on this first page was Child Health USA and, 
um, the reference guide says that's a title of an article, and gee. 
By checking field names on reference guide, he figures out 
fields, almost serendipitously. It's just thinking that, um, you 
know talking out loud is, uh, kind of confusing me. I was 
thinking if I had some peace and quiet I could figure out what's 
going on more. [Take your time.] Oh, OK. I'm going to go hack 
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and search, see if I can find a way to search, um, the subject 
not just obesity, but children and obesity. 

hE I'm going to try to figure out if I can, how to go back. [Beep.] 
Tries PREVIOUS RECORD from 1 of 228 to get back. Jumps to 
search field by clicking in window accidentally. I'm going to, 
um, type in erase. Now I'm going to try, um, the subject "child 
obesity". Hm. It says that, um, there's 228 records on obesity, 
19000 on child, 228 on obesity, 1 on um child obesity, so we'll 
see if I can bring up that record. 

? Um, let's see, um, it doesn't seem as though there's any, um, 
these lines, each line begins with some kind of code or 
abbreviation that, um, sometimes it's just not obvious what it 
means. 

(Jl 5//// checking reference guide for field names. Not familiar 
with concept of fields? Um, let's see. Since there were so 
many, um, records on children and there's only one on child 
obesity, I'm going to go back to obesity and, um, see if I can't, 
um, find, um. 

NE Still trying previous record and up arrow to get back to search 
window. Machine beeps, but no indication of what is wrong. 
Again, he gets back by accident. 

? From clicking in search history window, the search box shows 
"tflOBESITY". He gets error message and, without 
understanding the problem, erases it completely, and types in 
"obesity". [Keep thinking aloud.] Um, yeah, I've gone back to 
obesity and I'm going to start the search. [Beep.] And, um, oh, 
oh, I pressed the wrong thing. Here, try this again. OK, um, it 
looks like all these different lines that come up about this 
record, I think they're all about this one record. Um, let's see. 
It says that, um, it s?ys the title is Child Health USA '90, and I 
don't know whether that's the title of the article or a magazine. 
It sounds like it's the title of a magazine. 

Ql And the next code, CS, I can't find that. And the next code, RN, 
I don't really see that, um, hm. Refers only to reference guide 
for field names. I'm just going to try to write down some 
articles, 5 articles that I think most probably have something 
to do with children and obesity, and, um, since the first one is 
uh from a magazine called child health, and I searched on 
obesity, there must be an article in there on child obesity. [I'm 
going to interrupt you here to remind you that I want you to 
print several records out.] OK, OK, OK, let's see. Where's the 
printer? [Actually, it's over there.] OK, OK, um, I'm going to 
print, uh, if I can figure out how, this, um, this article, let's see. 
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How can I do that? Hm. There must be an easy way to get this 
printed out. Hm. No use of menubar until this point. 

UM Opens manuaU but sees thickness of it and closes it. Now 
checks under menubar items to see what's there. Hm. Sigh. 

H OK, I'm in, under help menu and I'm going to click Guide to 
ERIC and see what they say. 

M Trouble with secondary menus in Guide to ERIC. After several 
tries, selects getting help instead which has no secondary 
menus. Now I'm reading about getting help. Hm. Urn, I'm just 
going to bring up this article. Under file menu I saw where an 
option to print came up, so, um, I'm just going to print one copy 
of it. Um, the printer has been initialized with an incompatible 
version of the laser printer software. To re-initialize and 
continue printing click OK. Oh, it looks like somebody already 
printed something on the, um, printer, so it's got to wait. Um, 
I'm going to break the printing and search some more articles 
and, um, come back and print it when it's, when it's, when the 
printer is free. 

E He is printing all 228 records, but he doesn't know it. Break 
print doesn't respond, so he keeps clicking, then tries moving 
the mouse to the File menu, but gets no response there either. 
It finally does break. Hm. It is usually easy to cancel printing, 
but it doesn't seem to be easy now. I guess it's starting the 
printing. OK. So Fm going to move on to the next record, 2 of 
228. Um, it doesn't seem like there's anything about this type 
of title, this article having to do with children and obesity so 
Fm going to go on. Third one is not about children at all. Oops, 
5th one, um, oh, frequency for childhood obesity, now this one 
is definitely about children so I'm going to print this one also. 

E He doesn't read the print dialog box. Message gets up to 13 
records formatted before breaking print. Does he think it's 
printed? If printer were close by he would know. OK, my 
second article is being printed. OK, um, I'm going, um, to the 
next record, um, elementary physical education teachers' 
perceptions of childhood obesity. I'll go on, it seems as though 
there's quite a few on childhood obesity, and that one doesn't, 
oh yeah, the first five, I guess my instructions are to print the 
first five, so I'll go back to this. OK, and now I'm going to print 
this one, elementary physical education teachers' perceptions 
on childhood obesity. [Beep, beep, beep.] It says the printer is 
out of paper. [I can take care of that. Break in tape while I 
load paper.] 
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? OK, so there's this, um, strange thing at the bottom of the 
screen about the records formatted so far, and I don't 
understand what that means, but anyway, um. Tm going to 
make sure that this article on elementary physical education 
teachers' perceptions on childhood obesity is going to print so, 
we'll go, hopefully to get that printed, and then I'm going to 
find two more articles. 

E Break print again. Still formatting all 228 records. Breaks 

after 38 records formatted. Impatient with mouse, continues 
to click and look for other ways to stop print. 

? Oh, I see what this is. TI is the title and JN must be the journal 
it's in. But, again, it's just not really clear. I can't find JN in the 
list of abbreviations. Um, paper printer exit misfeed, um, yes, 
since the printer's not right here I can't really tell whether this 
is really being printed or not ^nd now I've got a message that 
one page may be lost. Um, I don't know what to do since the 
printer's not right here, um. [I'll go check the printer again. If 
. you'll just... Break in tape while I correct misfeed.] OK, the 
printer is being reset. 

? I wonder what this records formatting this is all about. Um, I 
think I'm just waiting for the printer. Um, oh, ok, it sounds like 
it's just starting the printing job of this third article. And, um, 
this machine can't go forward until it prints, so I'm just waiting 
for the printing. [Beep.] 

? Is it printing? [There are some documents printing.] Now 

there's another error in the, um, printer. Uh, and I don't know. 
[Just click OK '^n that.] 

S- Printer error has numbers 495-1100. Absolutely no indication 
of where to go to find what that means. [Let me ask you at this 
point, do you feel like you, that the items that you printed 
were, uh, sufficient to answer the question? It really doesn't 
matter if there were five of them. It's more a question of 
whether you found appropriate records.] Well, um, oh. Yeah, 
let's see, um, actually, now that I look at this a little more, it's 
not just children and obesity, it's actually about nutrition and 
exercise with children and obesity, so, um, the, uh, article on 
the, by the physical education teachers probably would be 
relevant to exercise and nutrition in childhood obesity, but 
these other two, let. me go back and look at some of these other 
things that I printed out. I can go back to a couple and see if 
they are actually specific, specifically about exercise and 
obesity. Um, um, hm. Yeah, this first article might not really 
be directly relevant to nutrition and exercise. Let me keep 
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going forward to make sure that, urn, I printed up something, 
this article here I can't even remember now what I printed out, 
which articles I printed out, but for sure this elementary 
physical education teachers article should be relevant to the 
subject of nutrition and exercise of obese children, Um, let me 
see titles of these articles quickly since Fm only interested in 
children's obesity and articles on nutrition and exercise. This 
one is about adults. 

Prefers searching 'manually' through 228 records. No idea 
there may be an easier way? If there was some way to limit 
this search to only children and obesity, I wouldn't have to go 
through all these 200 articles. 

Still not waiting for clock to turn to arrow, skipping records 
throughout. Uh, Fm on the 18th article and there's, nothing uh 
physical fitness as a mode for intervention in children. It 
doesn't say anything in the title that this is about obesity, but I 
assume it must be obese children because the search, I mean, I 
wonder how the search knows that these children were obese? 
Obesity is in boldface as descriptor. Descriptors list such things 
as children, nutrition, etc. but he is only looking at titles. That's 
right, I mean. Let me think. Nutrition, update infant, yeah 
infant's not really part of the subject. Childhood obesity 
related to TV addition. Um, hm. Hm, effects of physical 
educators' appearance of body fatness in communicating 
exercise concepts. That's not it. It's funny that the first one, 
those first few articles seemed to be about children and 
obesity. Hm. Well, this article doesn't seem to, energy 
expenditure walking with hand weights, doesn't necessarily, 
might not have anything to do with obesity. Let me go back to 
that one and see. 

Fm going to, um, see what select fields means. Seemingly out 
of frustration, clicks on a field name within Select fields and 
gets description of AN, but not helpful to him-it's not 
anywhere near what he wanted to know. Hm. Sigh. Accession 
number is just in the prefix. Hm, yeah, uh, keep clicking 
through the articles. 

Yeah, it's just clear, you know, how to use this, um, how to use 
the search. It's just not simple enough. Fve got all kinds of 
articles coming up that I really don't, I'm really not interested 
in for my search. I mean, I guess I would have to specify my 
search more, obesity and children. Fm going to go back, um, 
I'm going to go back and see if I can't, um, search a different 
subject, or make the search more specific. See this, um, this, 



uh, search history shows that there's only 1, um, article on 
child obesity, specifically. Um, Fm just going to go back and 
see if I can*t, um, uh, Fm going to go back and see if I can't find 
another subject to search. 

Clicks within search history, not realizing they go into search 
field, then backspaces to erase. Another way to describe this, 
um, I'm going to try childhood obesity. 
Why would there only be one article in child obesity and 21 
articles on childhood obesity? That just doesn't make sense. 
But anyway. [May I ask you a question here?] Yeah. [Why, 
um, if you searched child obesity, why then did you change it 
to childhood obesity? I mean, you did get more hits, but what 
led you to do that?] Yeah, um, I don't know. [OK, that's all 
right.] He read it in the descriptor field in a relevant citation, 
but I don't think it consciously registered. Uh, yeah, so, now if 
I can just bring up some of these childhood obesity articles and 
I still need to find 5 that I can use/ And, uh, then 111 be done 
with my search. Now, just, I wonder what the next step is 
going to be, Uh, I wonder if I can just search this by number, I 
probably can. Number 8. 

Used Caps Lock to try to get # sign. Hm, [Use the shift key aiKi 
3.] OK, I'm searching this by, um, uh, yeah, that's not working. 
Fm just going to type in childhood obesity and search that. 
Having trouble remembering to Show Records Found. Redoes 
search at least 3 times, expecting records window to just 
appear, I guess. Hm. Oh, [Beep,] Type in a search request 
before starting a search. Hm. That's strange, Hm. OK, I guess 
they just want me to hit Show Records, So here's that one 
record that I think was from my last search on child obesity. 
Child Health USA 90, So I got that one. So, you say I don't 
need to print this out? [Why don't you try to print one more.] 
OK, So let's go through, I'm going to go through these a little 
bit, and see if they're all, all, in fact, on childhood obesity. Ncii 
record, A follow-up of family behavioral treatments of 
childhood obesity. That's not actually necessarily nutrition and 
exercise, so we'll go on. 

Still not waiting for arrow. I don't think he's noticed clock icon. 
Is childhood obesity related to TV addiction, Um, going to next 
one, because I'm more interested in exercise and nutrition. 
This is from a sports medicine journal. Is there a crisis in 
youth fitness or fatness. Elementary principals' perceptions of 
childhood obesity. School nurses' perceptions of childhood 
obesity. Concerns of the physical educator. Now that one may 
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have information about how exercise and obesity arc related- 
Come on. Special reference break. Actually it would probably 
be better to search childhood obesity, nutrition and childhood 
obesity, exercise. If you could be that specific. Perhaps if this 
weren't a controlled environment he would continue to refine 
search— or give up in frustration, 

? Oh no, this is just printing up all these articles that Ym not 
really asking it. It's not printing at all. Why would he think 
it's printing? Hm. Sigh. 

L I think that any articles, um, that are about childhood, if there's 
only 21 articles about childhood obesity, I don't think there 
would be any other articles with any more specific information 
like childhood obesity and exercise or childhood obesity and 
nutrition, so anything, anything, any articles I need must be 
included uiider tiiis, this is a broader topic. Um, I'm going to go 
back and see if I can print, maybe go back and print this 
article, um. Fm going to print this article, prevention and 
treatment of childhood obesity. OK, so it's starting the printing. 

? Still wonder what this records formatting, here. I don't see 
what's taking so long to print, unless it's printing a whole, I 
don't know if it's printing a whole article or whether it's just 
printing this bibliographic information about the title and the 
journal and the year of the journal. 

E Appears to read print dialog box, but doesn't notice all records 
being printed. At least there are only 21 in this set. Again - 
21 records formatted and printed and he didn't notice. OK. OK, 
well I think I'm done. [I'll get you out of that.] OK. 

QUESTIONNAIRE COMMENTS: It was not clear to me if I was to print 

up references or not. Or, whether the entire article was on the disk 

or just a list of articles. There should be a description of how to use 

on the screen. 

Session 4 

This searcher has some Mac experience, but is just becoming 
familiar with it. She spent more time looking at the question 
than any others. 

I don't know what the question is exactly, so I think it's what is 
th^ current thinking on teaching young children about nutrition 
and exercise. It's giving me an indication of kindergarten 
through third grade. I want to try to find out what teaching 
means in using ERIC, what they might use as teaching. Are 
they going to use instruction or educational something. Um, I 
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might, I wonder if I should look it up first or if I should just try 
it fust and see if I get stuck and then look it up? 
TH Um, well let me look it up first and we'll see what the 
descriptors arc for teaching and obesity. Using paper 
Thesaurus. Will be interesting to see if she later uses Index 
online. OK, I'm not real familiar with using this. [You may 
have to speak up a little.] All right. [I know you're kind of 
thinking to yourself, but...] Educational courses, this could be, 
policies, educational counseling. I'm looking through here to 
try to find something that might, methods, that sounds good. 
Educational methods, teaching methods, practices, educational 
practices, related terms. Looking to see if there's anything else 
that might be a little more on target. What is the current 
thinking, I guess practices would be a good one. Oh, strategies 
might be good. OK, strategies looks even better. Oh, OK, it's 
telling me to go to educational methods, see more specific 
terms in instruction or training in the primary emphasis. So, I 
guess strategies rather than training or instruction. Interests, 
looking for instruction, but I can't really find it. Let's try 
training. Oh, I don't know if training would be it, because 
training could mean to become a teacher or something like that 
instead of to train others. Oh, experimental teaching. I like 
strategies so I'm going to go with that. Related terms and 
broader terms, educational planning, instructional 
development. Let's see if that helps me out. OK, do I have to 
tell you what I'm typing in? [Please.] I'm going to type in 
"obesity', uh, and, if I type it right, and I'm putting instruction, 
what is the term here, going to put the term in that is in here... 
oh, I just had it, "instructional developments". No problem with 
backspace key to correct typing. 
Sf Do you push enter on this or start search? Start search, ok. Oh, 
looks like it's going to be a lot of records because it's taking a 
while. OK. Didn't get any hits, I guess I was too specific, but 
obesity got 228. Let's try obesity, I think I'm spelling it right. 
Obesity and instruction. Retypes obesity, instead of clicking on 
#i. Obesity and instructional, because, why not obesity and 
instructional techniques or something instead of developments. 
OK, got 6 hits, um. With 6 hits, I could probably look at them 
instead of going through pre-school, kindergarten through third 
grade, since there's only 6 of them. Let's see, I'm going to look 
at the records and see if I'm far off on what I'm looking at. A 
checkerboard approach, a culturally oriented program, I don't 
see if it tells me what ages or anything on here. That might be 
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too specific if I put the ages. Uh, next record- Ho problem with 
mouse movement, does follow reading with mousse. Oh, but 
these seem a little outdated. Educating students about obesity. 
Urn, let me, let me look at some of the terms theyv niight give 
me here, like related terms in these records that Tnight help 
me. Wait a minute, let's go back to that last one. Practitioners. 
I'm going to start over. How can I do this? 
NE Trouble getting back, so checks under buttons, thnen file menu 
to restart -- doesn't look any further than file mnenu though. 
Select fields, show subset, search from record, hnm. 
Q Would the question mark help me? No. [Keep tiiiiinking aloud.] 

OK, Tm trying to find out how I can start this ovecr. 
f Oh,' click on any word or drag across any phrase that you want 
to use in a search. It will transfer it to the find rrecords dialog 
box. Click on any word or drag across any phiasae that you 
want to use in a search. Oh gosh. I'm really not used to this 
Mac, how to do it, I don't know. Dialog box? Letts see. Print 
records, restart session. Well, I wonder if that's the whole 
thing? I'm just going to restart it, and see if I caan start over 
because I got stuck there with something I didn't iiike. Sigh, OK. 
I'm going to try it again. Start it over. "Obesity' and, Tm going 
to put "teaching". I think I got too, I tried to do it the way I 
thought would be, the way I should do it, but I iiasually just go 
in and put any two terms in there. See what I ccome up with. 
13 obesity and teaching. Since there's only 13 raecords I'm 
going to take a look at them. To see if it's, all rigght, the effects 
of a physical education, this is a good article, but it's for high 
school students, so I'll go through 13 and see if ^y are related 
to pre-school to kindergarten through third. Oh. well looking 
through these terms I saw learning strategies amid obesity in 
here. That might be another way to look at this. I wonder if it 
would be a good idea just to use obese, instead oof obesity, but 
we'll see. Familiar with citation format, looks thnrough 
descriptors. This might be a good article, it says about 
nutrition. What's the current thinking on teachinng young 
children. Urn, please print out the first 5 records... I don't know 
if I'm supposed to look for the best 5 that I thinak are the best 
and print out one at a time, or if I find 5 all togeether print 
them. out. [Either way.] OK, well I think this womild be an OK 
article, so to print the record, I'll go up to print lat, and see what 
it says. Citation only will be printed. I think thaat's all we need 
is the citation. OK. Push print to print it. Urn, loooks like it's OK. 
Zero records formatted so far, click the button too break. 
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Finds print under file, but doesn't notice it will print all 
records. Does notice that it prints all .13. Oh, I guess it's 
printing all the records I found, all 13. I thought it would print 
the one that was on the screen at the time. Well, I'll print all 
13, there might be another one I like in there. OK. Go to the 
next record and take a look at it. See if I like the next record. 
Looking at the next article, see if I like it. Got a lot of articles I 
don't really need from that print. I could probably use that 
last article, so that would be two. Um, not all of them tell you 
ages that the articles are related to when it comes to guides to 
classroom teaching. I would probably like to know if there's a 
way to, in the source, to tell you. Descriptors do indicate 
educational level and terms hyphenated to ease search from 
records, but you often have to click the down arrow to gei 
them into the window. That one says junior high, that's not 
good. OK. OK. 

Restarts session again here. Show subset, select fields. Oh, 
show subset. That might tell me how to get back. Oh no, cancel 
that. Select records, search from records. 
Click on any word and drag it across. I just don't understand 
what this means. Click on any word and drag it across any 
phrase that you want to use in .i search. It will be transformed 
to the find record diagonal box. Search from records 
explanation confusing, particularly because she misreads it. 
There are several Mac terms here that she is not familiar with. 
Where do I drag it? Click on any word and drag it across any 
phrase. That's probably how I start over. But I don't know 
how to. Restarting a session. [Keep thinking aloud. Tell me 
what you're doing.] Ah, I'm looking maybe to start over, and 
then I saw this search, if you read it it would probably help 
you out refining search, but I'm just going to start it over again 
and use these terms that I think are better, looking through the 
journal articles and they tell you some terms that were used. I 
think if I was going to do this I would have read the 
instructions before I started. Let's use learning, let me put a 
dash, strategies and obesity and, let's see, um, and I'm going to 
put elementary. Try it this time. I got one article. Probably 
too specific, maybe. There were only 228 obesity articles. I 
should be able to get something out of that one. Let's see, show 
record, one of one. That's the same article I got last time using 
those terms. Oh, let me just try elementary and obese. 
Restart session. Well, I have two articles. [Why don't you try 
to get one more.] OK. I've failed miserably. [No, no.] And, um. 



elemenlary. That's a lot of records for it to look through. Oh. 
OK, physical education, oh this is a good article. Can you print, 
let me look at the print to see if you can print just a certain 
record. Print records, options for print, highlighted hits, line 
long short none. Highlighted hits? I don't know how you 
highlight a hit on here. Print records, save records, I mean, Fm 
looking up, using print records, select fields, select records, 
page setup, options for print. Sigh- 

H Checks under help menu, but doesnt select secondary menu 
that would have answered question of how to print selected 
records. Wow, I don't want to print them all out, but I like 
that. Let me look up print records, it might give me an option 
to print, like 1 to 5 or something. Oh, selected records, oh, but 
that's not highlighted so I can't do that. Oh, start printing from 
record 1, OK, let me cancel and see if I like the other ones. Ok, 
these articles seem like they're a little bit better on taiget. 
Obesity wasn't really necessary, I don't think. Just physical 
education and young. 

E I'm going to print the first 5 because I need the last one. Do I 
need this one? The first 4 I think it was. OK, the first four. 1- 
4. I guess that's how you do four- [Beep,] [Keep thinking 
aloud.] OK, I put 1-4 with a dash, it doesn't want to do that so I 
guess, how about if I type to 4. 1 to 4. [Beep.] It doesn't want 
to do that. Darn. Backspace again, print 1 comma 4, maybe 
that's it. [Beep.] 

S- Lots of beeps, but no message to direct her to select records. It 
didn't take the dash- Oh, well no, start printing records from 
number 1- Let me look at this little help- Is that a help 
button? 

Q Does click ? icon, but expects context-sensitive help to just pop 
up. Gives up. Hm, I don't want to print all those, all those 29 
of them- 

H I'm going to go up to help to see if it can tell me something 

about printing records- No, it doesn't really say anything- Print 
records, use print records, selecting fields, selecting records- 
Mac [Since you're new on the Mac, let me help you out here a little 
bit- Go back under help- And come down to printing records, 
and that's a secondary menu, if you'll move the mouse over, 
and then select one of those-] Oh, OK, selecting records- Do it 
by clicking on any record number, in other words 3 of 28- 
When you click on a record number it is displayed in bold- Oh, 
OK, maybe I have to select the record- Do it by clicking on any 
record number, in other words, 3 of 28- When you click on a 
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record number it is displayed in bold. [Keep thinking aloud.] 
Oh, Ym reading it to make sure I understand what's going on. I 
don't know if it's telling me to click SELECT RECORDS every time 
I want that one to print. Let me try that. I have to close this 
window. Talk about being a rookie. It's, SELECT RECORDS. 
Select records by clicking on a record number, in other words, 
1 of 10, Selected record numbers in bold type. Click again to 
deselect a record. It is not necessary to click the Select Records 
button before selecting a record. Oh, but where's the record 
number on here? Is that? Oh, OK, so that's how you do it. I 
wonder if this goes back. I guess not. Uh-uh. Let's just go 
through these records. I'm clicking on SELECT RECORD. I think 
that's how I can get it to print these. I don't want that one. 
This first one. OK. Now, let's see if I can print it. See what it 
says. Oh, OK, selected records are on there now. OK. So it 
should print my selected records now. It's just 4, or 3 of them 
now. Well, I caught on. It took me a while. That's 5. [OK. We 
can stop here.] You going to teach me on this now? 
QUESTIONNAIRE COMMENTS: The system seems to have a lot of 
capabilities, but I didn't know how to use them. With some 
instruction and practice with the computer itself I ihink I would be 
much more successful. Hence, my opinion * would change from 
frustrating and discouraged to satisfying. 

Ses$i9n g 

Mac experience, has a Mac at home. Search experience on ERIC, 
LISA, Library Literature, but all on PC, 

OK, well, first I would search, I think, under obesity. Obesity, 
and then, oh, how do I do this? Type a search request and click 
start search. Where? Oh, there it is. OK. Now, this gives me 
my records immediately. Like, the other one, you know, it can 
keep going and gives you like a little time thing. This one will 
just bring it up immediately from the looks of it, I hope. And 
next I'm going to look for young children, er, uh, young people. 
I'm going to look under adolescents. Oops. I hope I spell it 
right, adolescents. Ha, I just spelled it wrong. OK, let's see, 
what, OK, um, what is the current thinking on teaching. Sigh. 
I'm going to combine one and two. 

Types 1, clicks AND, types 2. System interprets as #/ and #2. 
One and two. See what I come up with. Ah, good, I got 11 hits. 
OK, um, and what is the current thinking on teaching young 
children about nutrition and exercise. Um, I do have a question 
about the search question. Is this all concerning one, or is this. 
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I mean Fm concerned with obesity in young people, um, what 
is the current thinking on teaching young children about 
nutrition and exercise. Uh, this is actually what you want, this 
is what your concern is, this is what is prompting you to, this is 
what's prompting you to ask this question, [Yes,] OK, OK, OK, 
young children, preschool, kindergarten through third grade, 
nutrition and exercise. I think I'm going to search nutrition 
and exercise. Nutrition, ooh, I can't read, [I'm going to ask you 
to speak up just a little bit,] Nutrition, OK, exercise, I spelled 
exercise wrong. Exercise, Um, OK, I'm going to try it, I'm going 
to combine 4 and 5, 4 and 5, No hits. OK, um, hm. Nutrition 
and exercise. How about let's try. I'm going to try, just for the 
heck of it, call these up and look through them real quick. 
Performs several single word searches, then combines with 
numbers. Except for I don't know how to do that. Well, let's 
try it this way. I'm not familiar with this. SHOW RECORDS 
FOUND . OK, that's what I want to see. 11, This one is younger 
children. See, I thought young people, I put adolescents, but 
you're really talking about young children, I think I'm going to 
go back. Previous record, next record, select field, search from 
records. Oh well, maybe next record. Determinants, These are 
too old. Previous record. Oh, select records. Let's see, where 
do I want to go back? 
NE How do I get back to where I was? Previous record. Come on, 
S- [Beep, Beep,] What's that? Search from records. Select fields. 
No, I don't want that, OK. Huh, OK, When trying to go back, 
looks under buttons, then tries previous record until Beeps. 
Finally closes window to get back. OK, preschool, sigh, let's see, 
K-3, let's try, preschool, oops, preschool, what am I doing? Um, 
huh, I don't know, can you search, like, K through 3? I don't 
know if you can do a search like that? [You can try it,] K, I 
don't know, I probably doubt this, K through 3, I don't think it 
will know what it's talking about. Grades K through 3. 
I keep wanting to hit that return button. / don't think she's 
noticed that she has alternately hit the return key and the start 
search button, and they both work. Huh, well, great. Oh now, 
here's where it will say. OK, it came up with a big zilch, OK, 
let's try, OK, let's try, how did I combine those before? Number 
6? Yeah, 4 and 5, Number 6, um, and 13, Zilch, OK. Um, hm, 
[Keep thinking aloud,] What is the current thinking on teaching 
young children about nutrition and exercise? I'm going to try 
young children, what the heck? Young children. Young 
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children. Now, Ym going to try 17 and whatever that was. 17 
and 6. Still going. 

L 6 and 13, 17 and 6. 6 ncd no hits, so she'll get no hits on any 
combination with 6. Hm. OK. So let's do young children, 17, 
and number 6, yeah, number 6. Sigh, big fat zero. OK, urn, let's 
try, nutrition. Hm. Let's try education. Education and 
nutrition. Education and nutrition. This is what I mean where 
I sometimes, I just go for the obvious and then after that Fm 
not sure. You know? Uh, that's why I don't think I utilize it 
the way, you know, completely. Oh my god, it's only 17. Only 
17% complete. Come on. Obesity. Sigh. Oh, god, come on. I'm 
going to quit there. Breaks search at 726 records. Um, OK. 
Well let's try 21 and 17. 21 and 17. Hm. So far a big zilch. 
Education. Hm, OK, try teaching and children, teaching, 
exercise. Well, it looks like we're coming up with a big zero. Of 
course, I always hold out. I must be searching under the 
wrong thing, because it certainly seems like there would be 
something, material, you know, on educating young children on 
nutrition. Hm. OK, let's try, we're not finding anything on this, 
go on with this line of thinking. Um, OK. Let's see what else we 
can do here. Um, nutrition, exercise. Well we're not going to be 
able to teach them anything. Let's look'^under teaching. 

TH Looks at Thesaurus. How does this thing work? OK. Teaching. 
Education. Oh, grades, higher education, high school. Let's look 
under, hm, I don't know how you, grade, grade, OK, here we go. 
Elementary education. Sigh, let's try elementary education I 
guess. I don't know. Elementary education. Oops. Elementary 
education. Let's see. Let's try 20, 25, I'm sorry. Now, I don't 
know what these things are. What does this stuff mean? BT, 
broader term, related term. Oh, OK. Checks front of thesaurus 
for meaning of abbreviations. Primary education. Oh lord, this 
is going to come up with a zillion. 

L Breaks at 8000 records, but then combines with #6 which had 
no hits. OK, let's try 25 and um, gee, what was that one, 6, I 
think. Instructional program division. Looks like I'm not going 
to find anything there either. OK, what is the current thinking 
on teaching, hm. How about teaching? Another way to say 
teaching. Teaching. Teaching methods. Teaching guides. 
Teaching. Instmction, teaching methods. Oh, teaching 
methods, maybe. Let's try that. Oops. Teaching methods. 
And, let's try that. Probably come up with a big zip again. 

L Again uses 6 in combination with teaching methods. Teacher 
placement, no, no, no, no. Teaching methods. Let's go under 
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teaching methods here. Ways of presenting, search on 
activities, integrated methods. Ooh, integrated methods. 
Subject methods. Oh, I wish I knew more about teaching. Um, 
hm, creative teaching. Educational methods. Classroom 
techniques. Instruction? Teaching models, no. Probably come 
up with a big fat zero there. OK, let's try. What did I see here 
that looked good? Um, current thinking on teaching, current 
thinking on teaching. There's a word in here, but I'm not 
getting it. My head is full of cold. Um, educational strategies- 
How about that? Let's try that. Educational strategies. Um, 
oops, spelled it wrong. Strategies. No, strategies. Let's try that. 
Educational strategies. I'm sure this is going to- come up with a 
million. Sigh. I'm not real good at narrowing it down. You 
know, I could always figure out the, um, you know, sort of like 
the key search words, but then when it comes to, you know, 
like, the whole, you know, like, the current thinking on 
teaching, then that's where I am unclear on how to search, or 
how I would search, so it's just hit and miss. Obviously. Ha, ha. 
OK, let's, I hate to quit these because it's, like, I figure, like 
when it gets to 75% it might be the one I want, you know. 
OK, education strategies, teaching guides, no, teaching models, 
no. Teaching practices, no. No. Let's look under, I don't know 
if that would give me anything. Look under nutrition, but I 
don't think it's going to, I don't think it goes that way. 
Nutrition. Nutrition, nutrition, see if there's another way to 
look for nutrition. Again, while waiting for search to complete, 
she is perusing thesaurus. Ah, health education. I like that. Do 
you like that? /'[Ha, ha.] I kind of like that. Let's quit this. 
OK, let's try, uh, OK, let's try 32 and my good old number 6. OK. 
I'm sure it will come up with zip. Something, something's not 
jiving between those 2. I don't know what it is. It seems like 
that would be a good indicator, those 2 together, you know. 
Teaching methods and nutrition. Well, wait a minute. I have 
nutrition and exercise. What's number 6 again? Diet, nutrition. 
Nutrition and exercise. [When you break here, why don't you 
scroll up and look at number 6.] OK. Crud, I'm not finding 
anything again. Health education, I like that. It's a little more, 
clued into this. All right, forget it. I'm giving up. Yeah, 
nutrition and exercise. That's number 6. OK, let's go back to 
where we were. OK, let's try health education. When she 
scrolls up to look at 6, it's a combination of 4 and 5 - nutrition 
and excercise - 4 hits on this misspelling of exercise. Health 
education, health education, health education, and, I wonder if 
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I'm using the wrong terms, combining tennns, or health 
education or, another term, let's get rid of that Health 
education . and #6. 

L Still combining with 6. Didn't notice logic error of using line # 
with no hits. Well, OK. Quit it. OK, I waant health education, 
let's see, number, OK, 36 and, where do I :liave, grade? K-3, um. 
13. Hm. So I want 36, 36 and 13. [Can : I interrupt you for a 
minute here? You have a flaw in your loogic that's really 
hanging you up.] OK, that's what I was, yvoo know, I know 
there is, but I just don't know what. [Thins column indicates 
how many records were found. And anyimme you combine 
something with zero hits you're going to gect zero hits.] Exactly. 
[So, for number 6, you had zero hits ondeer nutrition and 
exercise, although if you'll go back, youH sec that on a previous 
search you misspelled exercise, but you stillll go hits on it.] Oh. 
[But, what I want you to notice here is thaat any time you 
combine something with zero hits, you're ^always going to get 
zero hits.] Exactly. OK. So don't even usee this number 6 in 
other words. OK. Now, let me go back thearre and look. Oh god, 
I didn't even look at that. How smart- Umn, K-3, see there won't 
be anything there either. Young children, well let's try young 
children, 17. 17, and, what was the other oooe? Education and 
nutrition, 21. Dam. That's what I hate abooot logic. So logical. 
Oh, come on. Sigh. 

TH Um, nutrition, eating habits, oh, nutrition instruction, I like this. 
OK, forget it. Break search at 16%, zero hhits. because she finds 
something in thesaurus she likes better. Frrustration? Or 
general impatience? Nutrition, nutritioiL when I use two words 
like this, that's OK, right? Instruction. I rnnean, I can do that? 
[It appears to be working.] OK. I mean, at has two words here. 
It just puts them together? OK, nutrition instruction. Oh, thank 
god, I'm getting something. OK, nutrition instruction. Let's see, 
nutrition instruction. Now, how do I get esxercise. Let's see if I 
can get exercise here? Exercise. Oh, Tm -more clogged up than 
I thought. Exercise, exercise, exercise. Hmn, I don't like any of 
those. OK, nutrition instruction, I got someetfaing at least. Um, 
what do I want? Uh, 41 and, do I have yvoung children u;p 
here? Yeah, young children, 17. Let's try that 

E Oh, poop. It looks like we're going to get big zilch here. Crud, 
I can't believe this. She actually got 17 hiiits on 41 and 17, but 
didn't get them until the last percentage of ' search. OK, let's go 
back and see what we've got here. Educsation, nutrition. 
Obesity. Sigh, obesity and young children... Let's try that, what 
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the heck, it can't hurt. Obesity and young children. 17. OK. 
After I'm done with this, will you tell me the error of my 
ways? [Yes.] Thanks. She had expressed frustration before 
this session began over a search she was conducting for a class 
where she was not having any luck. Logic and search 
strategies would be good tips for her. Sigh. I know there's, urn, 
there's strategy to search, I'm just not, I haven't used it that 
much to, to um, to play out those strategies first. I've just used 
this a couple of times. Oh man. OK, something's wrong here. 
I'm using the wrong, uh, oh, wait a minute, I got four. Let's see 
what we've got. [inaudible reading of citation information] Can 
I get a print of this? Either she didn't read the instructions 
carefully or she's totally forgotten the instructions after her 
search difficulties. [I'd like you to print, say 2 or 3, but no 
more than 5.] OK. All right, how do I print? Print records. 

E Easily goes to File menu, changes to start from record #2. It 
printed 3 of the 4 records, but she didn't notice. I just want 
record 2. Now, OK. OK. 

? What does this mean? Zero records formatted so far, click the 
button to break. Oh, it started, OK. Um, let's see what it says 
here. Early childhood, nutrition, physical activity, physical 
development, eating habits, physical education. I want to get 
that one printed. OK, that looks like a fairiy decent one, too. 
I'm going to do that. Print records, I just want number 3. 

E Selects start from #3. I want, oh, OK, I want, 3 to 3. That's all I 
want. In LaserWriter dialog box selects print from page 3 to 
page 3. Still prints 2 of 4 records. OK, um, OK, I don't like that 
one so much. OK, let's go back. Um, can I get that copy? [Uh- 
huh. If you feel like you've printed a couple that are good, um, 
to answer the question, that's fine. You can stop if you like.] 
OK. I guess, um, I want to look at -some of the other fields they 
have and see how to combine them. If there's a better, 
something, you know, because I want to get the teaching 
methods down. [OK.] Here I go to get the printed records. OK, 
let's go nutrition instruction and obesity. Nutrition instruction. 
[Here you go.] Oh thanks, let's see. 

E No, shoot, this doesn't tell me the whole thing. Retrieved 

printed records, but they don't include descriptors. Because I 
want current teaching, I want to get that in here somehow. 
Um, teaching methods, educational strategies. Let's try that 
again. Educational strategies, 32, and, um, uh, 41. Sigh. 
Educational strategies and 41. Let's see, let's try that. Forget it. 
Continues to break search before complete. OK, let's try. 
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educational strategies and, um, obesity. Educational strategy 
and fat kids. Gosh. Nutrition instruction. Eating habits. Oh, 
I'm not finding anything. These seemed so good, Objectives. 
Goals, objectives. Teaching nutrition. OK, forget it. Forget it, 
forget it. OK, let's go for, OK, what's the current thinking on 
teafching young children, teaching young children, uh. I don't 
know. I'm going to just, primary education and, nutrition? 
[You've printed several records. If you are ready to stop, I 
think you've completed the assignment. But it's up to you. If 
you'd like to keep trying to narrow your search...] I just want 
to try one more thing. This is making me mad. Nutrition, 
nutrition, OK. Oops. Exercise. All right, now let's try something 
else. And, and, OK, let's try primary education, 38, with 
nutrition, 49, and, exercise, 50. And I'm quitting after this if it 
doesn't come up with anything else, because I don't know what 
I'm doing wrong. Learning processes, ooh, that's good, learning 
processes. Well, it looks like we're not going to find anything. 
See I'm not sure how you, um, yeah, I'm not real sure how you 
put all this together. With teaching young children. All right, 
forget it. I'm not going to find anything. OK. 
QUESTIONNAIRE COMMENTS: None. 

§^§§iQn ^ 

Very little Mac experience, not comfortable with the platform. 
Made notes on page from own ideas then checked quick 
reference card 

TH and thesaurus under nutrition (2 matches: obesity and 

nutrition information). "I've put it off long enough, it's time to 
start", she says as she turns to the computer. Just glanced at 
the screen before going to the reference guide. 
[Ail right.] Problem with tape on this interview. She first 
searched nutrition instruction and children. She was a little 
hesitant with the mouse, but she reaches for it readily to click 
START SEARCH. No problem finding Show Records Found 
button. [OK, pick up right where you left off.] OK. You are 
what you eat does not necessarily deal with educational 
practices. We'll iry again here. Cooking in the classroom. We 
have 99 hits on this search. And, I think the problem here is 
that it's much too broad. I've got to zero in on obesity 
somehow. Part of the problem here is figuring out, of course, 
which is why you're having me do this thing, what the correct 
combination would be to narrow it down. Um, one of the things 
I'm not sure about because I've never worked with ERIC before 



£RIC 



101 



93 

is whether I can combine moore factors in the search. In other 
wbrds, Fve got nutrition instzmiction and children. Uh, Ym not 
sure whether I can do nutrimion instruction and obesity and 
children. Uh, or the other coompooent of this was exercise. Uh, 
but I think instead of whippmng through 99 of these things, 
which would take forever, dmat I need to try something else. 
Now, I have to figure out how to do that Hm. Ha, ha. Which, 
of course, would be kind of ihandy if I knew how to work the 
Mac and I suppose I could goo to the guide here. But, well, 1*11 
just try getting out and see 'what hsq)pens. 1*11 probably louse 
up the whole thing. 

^E Clicks apple on menubar whenn trying to get back. No, Fm not 
going to mess with that. How about SELECT FIELDS again and 
see what happens. That's noc: going to help me either. Back to 
Search and see what that doess. Click on any word or drag 
across any phrase that you waant to use in a search. Sigh. 
That*s not going to work. 

H Let me go up to help and seetr what that says. Bypasses 

secondary menus in help, to go down to restarting a session 
which directs her to File mennu. Oh, restart a session here. Sigh. 
On the file menu. OK. Wc haave the little clock, oh here it is. 
She waits patiently not naervously for restart. I'll try another 
search and see what happens^. Whether we get a better 
representation here. [Tell mee what you're typing in.] Trying to 
do nutrition instruction and cobesity and children. And see 
what happens. Last time, widthout the obesity in there, I got 99 
hits. And they were all over the map. A lot of stuff, basic 
nutrition in school, you art what you eat, that kind of thing, 
which doesn't zero in enough... Now we still haven't hit exercise 
or physical fitness. 

? What happened, it went off :ihe screen here, it doesn't show 
anything. [Keep thinking alomid.] Search completed, found 5 
records, but she doesn't remaember to Show Records Found. 
Well, I have no idea what jmist happened. It was showing the 
percentage searched, and tbenii all of a sudden there was 
nothing on there. I may haiive done something with the mouse. 
Probably did. Uh, it's not daoing anything at all. 

Q Therefore, I'm going to go ower here to the big black diamond 
with the question mark in it and see what happens. Nothing 
happens. Ha, ha. 

Nf Clicks on ? icon, the re-clickxs to get cursor arrow back. All 
right. It's showing me to da the enter search again, at least 
that*s where the little marker^ is. So there was something that 
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didn't work out on that one. Um, if I just go to obesity and 
nutrition instruction, and forget kids for a second, ws're going 
to probably get more hits again. Um, but wait a minute. Up 
here it was showing that there were 5 records on nutrition 
instruction and obesity and children. As opposed to 40,000 
plus for just children, so, I guess all the blank means is that it 
fmished the search, and I just wasn't paying attention. I want 
to show the records found, but I just want to show for number 
5. Hm, well let's try this and see what happens. OK. All right, 
childhood obesity, concern for the physical educator, that 
sounds good. That does the trick very nicely. Let's see what 
the next one is. Intervention for obesity, current approaches. 
I'm looking right now to see whether it deals with phys ed as 
well as nutrition. 1 out of 2 so far. Overweight children and 
phys ed. There. I think we've got our hits at this point. Um, 
let's try the 5th one. Yeah, those look very good. And, I think, 
let's see, my instructions here are to go ahead and print 5 
records, and I've got 5, and I'm just going to go for it. As soon 
as I figure out how to print. So, um, now what? It's probably 
one of these functions. I remember that from, there it is, that 
ought to do the trick. Select fields, all records, start printing 
from record 1. Since we're just printing the ones I retrieved 
and I only retrieved those 5, I don't have to worry about the 
other 40,000 plus that showed up on thb initial search screen. 
So it's going to the LaserWriter, and one copy. Looking for 
LaserWriter. Oh, OK, they found it. Got the 5 formatted, and 
I'm assuming that it's printing out. Somewhere, on the other 
side of this screen. OK, then that should be it. Prints total of 5 / 
records retrieved, so no need to select records. 
QUESTIONNAIRE COMMENTS: My lack of familiarty with Macintosh 
limited my enjoyment of the exercise, but I liked the logic of the 
system, its speed, and the ease with which it progressed from one 
function to another after I had figured out what to do. 

Session 7 

No Mac experience. Not familiar with ERIC. 
[You May clip this mic to your jeans or leave it in your lap or 
on the table, whichever is more comfortable to you.] OK. OK. 
In order to, um, search on obesity in young people, since I'm 
not sure what they mean by young people, I'm going to start 
with ct>esity and children. 
M Readily reaches for mouse, but a little slow with moving it. 
And the results for that was 68 records for obesity and 
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children. [Keep thinking aloud.] OK. Next, Fm going to try 
obesity and adolescents. No problems backspacing to correct 
typing errors. And there arc no search topics, or no records 
found for obesity and adolescents. So we*re mainly stuck with 
the 68 records on obesity aiid children. So, well see. [Keep 
thinking aloud.] We probably should go ahead and display 
what the 68, at least part of what the 68 records say about 
obesity and children, at least some of the headlines. How do 
you display the records after they are found? [Are you 
familiar with the menubar on the Mac at all?] No. 

Mac [If you'll move up, for example, to Help, and hold the mouse 

down. It will show a pop-up menu, and if you'll come down to 
Showing Records, there will be another menu here on the right, 
and if you bring the mouse over and then lift the clicker, it will 
give you instructions on that. Each one of these items will have 
a menu underneath it that you can bring up.] Oh, OK. 

H Familiar with close window, from help window, so she may not 
be as much of a Mac novice as she says. OK. [Keep thinking 
aloud. Tell me what you're doing, the logic you're using.] OK. 
In order to show the 68 records for obesity and children, we 
needed to go back and retype the search again, ind now Tm 
looking at the retrieved records, and, looking to see what, some 
of what, wm, the records have to say about obesity and 
children. Oh. 

Nf Uses down arrow to review record, to exclusion of Next Record 
button. [Keep thinking aloud.] OK. I am still looking at some of 
the records to see what they have to say about childhood 
obesity and if, and if most of these are what the, um, person 
who asked the question really needed, if the information is 
actually reflective of what my search strategy was. I'm hitting 
the SELECT RECORDS bar. [Keep thinking aloud, what are you 
looking for, v/hat's 3'our next step.] Since most of them tend to 
have what we need, it might, I don't think it would be 
advisable to go ahead and limit the search because most, it 
tends to be pretty limited on its own. [Keep thinking aloud, are 
you looking for help on printing records?] Yeah. Well, we'll 
just go ahead and get out of looking at this and look for the 
nutrition and education and children. 

NE How do we just get out, out of this application? Or do you have 
to print? [You should find some guidance there in the help. If 
you go under Finding Records, and maybe Narrowing a Search 
or Clearing Searches or Lateral Searching, there could be some 
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ideas. Or yoa comiid check under the different menus here and 
look under and see if any of them give you some ideas.] 
H OK, I've gone ondccr the Lateral Searching in the help bar. 

Typical trouble gemting back, perused help and viewed a few 
secondary menus. I tried to be as vague in order not to 
directly answer her- question, but she is clueless as to what to 
do next. No attemnpt to use paper sources. No attempt on her 
own to look at ? iccon or at help even after I first pointed it out 
to her. This is a wery passive searcher. I bet on her own that 
she would give up ^and go to paper - or seek help? [Keep 
thinking aloud.] Well, Lateral Searching isn't what I need. OK. 
[inaudible, pennsiag: help.] [Keep thinking aloud. You're 
Clearing a Searck] So, we'll go. ahead and search with nutrition 
and education and children. And see what that brings up. 
There were no reccords for that, so we'll just try nutrition and 
education. 

E Broke search of nuutrition ' and education and children saying 

none were found bout search hadn't finished. OK, there are 100, 
possibly more than 125 records for nutrition and education. So 
we are going to haave to limit that, however, nutrition and 
education and childrtren is not listed. So, [keep thinking aloud.] 
so, hm, let's see. 

TH I'm going to look inn the EPIC, er, ERIC Thesaurus to see if it 
might be, if there s a different word that I should use instead 
of just children. We'll go ahead and use kindergarten and 
children and the nnutrition and education search from before. 
See if there's an>tlihing that comes up with that. [Keep thinking 
aloud.] 

E And with the seanxh being 34% complete with no records, I 
doubt if there is amiything there under that heading, too. So, 
[I'm going to interrxupt you at this point. Rerun number 4 and 
let the search corapDlcte all the way, instead of breaking it 
before it's found reccords.] OK. Knowing that searches have 
only been showing records near 100% completion, 1 was afraid 
this could go on foorever. Researcher impatience a minus in this 
type of study. [Teell me what you're thinking at this point.] 

RT Well, it took quite .a while for it. [Was that frustrating or did it 
make you nervous that it wasn't going to find anything, or?] It 
did, because, Tm unsed to having things come up automatically. 
I'm not used to wamting until a search is over 25% complete to 
have it come back .and tell me that there's anything in there. I 
realize that might hoc just a time lag, but, like, I made the 
wrong assumption when nothing came back. I'm not used to 



ERIC 



having that in there. So, I think that's one of the limitations of 
this, if you're not used to have something like that, and having 
to, I'd rather have, have the, urn, have some indication that it's 
still working, but yet, you have to know that there's at least 
something in there on it. Sigh. Well, there were 496 records 
found for nutrition and education and children, which is a lot 
better than the zero I got before. So that makes me wonder if 
there isn't something on teaching kindergarten children 
nutrition and education also. And we just need to wait until 
the end of the search to find out. Because we only stopped at, 
uh, 34%. [So you're going to rerun that search.] Yeah. And just 
let it go all the way out, for a more accurate count. Since I now 
know what the database, er, the program does. This subject is 
unusually quiet. OK, well, now it's confirmed there are no 
records under kindergarten and children and nutrition and 
education. OK, let's see. 

Consult the trusty thesaurus again. [Tell me what you're 
thinking.] Find out, maybe, elementary aged children. Hm. 
Try elementary, education, oh wait, elementary and education 
and nutrition and education. See what that search brings us. 
While that's trying to find it, I think I'll look to see, there might 
be something under elementary school curriculum. Let's see. 
[Keep thinking aloud.] Well, since the person wants current 
thinking, and we're getting a wide number of records, you 
might want to limit using publication year. That way it will 
allow us to get more current ,being defined possibly as the last 
year or two. 

While searching elementary and education and nutrition and 
education, looks over quick reference guide about limit by 
publication year. 

As the search grinds on, she taps the table impatiently. [I 
didn't mean to scare you from breaking a search. If you feel 
like you have too many, and you don't want to wait anymore...] 
No, that's OK. I just didn't realize that was, urn, the way this 
database works. I'm also wondering if elementary education is 
just a little broad. 

Let's see what it does if we put, let's see, publication year 1992, 
go ahead and say publication year 1992 and the previous 
search which was elementary and education and nutrition and 
education. PY-1992 and 18. And, we come up with zero. 
Database goes to September, 1991. This is underneath the 
menubar area, but she hasn't read it. OK. [Keep thinking 
aloud.] Let's see what publication year 1991 is going to say. 
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along with elementary, nutrition, education, elementary 
schools. These are not new search terms, she just says them 
differently. Yawn. [Keep thinking aloud.] Hm. Well, there's at 
bast some. And, I find it somewhat interesting that, either 
there wasn't anything written about nutritional education in 
elementary schools this year so far, or since there's been all the 
recent advertising for fitness in schools this year, I just find 
that interesting that there's zero here so far. 

E I'm going to wait until this gets up to maybe 50% and then 
we'll break it. OK. So there were four records found for 1991 
for elementary education and nutritional. I think that 
elementary education would cover at least kindergarten 
through third grade. It might be a little more. [Don't focus on 
the fact that you have to get five. If you get less than five and 
you feel they're appropriate, that's fine, you can print those.] 
OK, so you wanted five from each section then. [No, just if you 
get some, I'd like you to print several records, no more than 
five.] OK. [But if you find four, that's OK. It all depends on 
how your search goes.] Right, OK. Well, before I print anything 
I jus! want to go on to education, er, exercise first. 

L And I'll just put exercise and education and elementary and 
education. Since this search request was also for nutrition and 
exercise, we might as well see what the database has for the 
exercise education and elementary. OK. OK, the search at about 
25% complete and there are 128 articles, so we'll limit that 
with publication year equal 1991 again. See how that works. 
OK. Let's see, at 30% search completed we got 9 articles for 
1991 publications of exercise, education, elementary, and 
education, so at this point, it would probably be best to look at 
some of the articles that we have pulled, or some of the records 
we pulled up. The first one will be nutrition, 1991 publications 
of nutrition education and elementary education. I'm going to 
SHOW RECORDS FOUND. 

RT Sigh. It's taking a long while to pull up 4 records. I wonder 
what the problem is. 

S- Message is: Retrieving record, please wait. It is taking a long 
time. No indication of why or % complete. She's very 
frustrated. Sigh. I'm thinking we should keep narrowing it 
down, since you have to put the phrase, you have to re-type, in 
order to use previous records, you have to start the search over 
again by hitting the line number from the windows menu, so 
number 4, SHOW RECORDS FOUND, and we're getting this 
message which isn't helpful. This isn't a good one for an 



impaniieat person. Ha, ha. Hm. [If I weren't sitting here, what 
wouldd you do right now?] Break it, and go on to the next one, 
at Icaast, although the patron really wants it. I'll go ahead and 
break, it and see if the same thing happens with the, urn, 
exeiccise one. And for this one we'll have to wait because we 
did a. complete search the last time. In order to. Sigh. [Why 
don't you go ahead and break it and see if you can show some 
recoraris from number 3.] OK. [Why don't you print, say 2, any 
2 rcccords from thai search.] OK. She read Select Records 
inforrmation. Again use of arrow keys as opposed to Next and 
Previaous buttons. Finally does use Next. Does highlight 
documneni #. Use of arrows delays cursor return to arrow, she 
shows' s some impatience there. OK. The, um, record that I 
selcctaed that was a conference paper having to do with various 
healthn education topics for the elementary education group, 
and int also had to deal with physical and exercise. [Why don't 
you purint one more.] OK. Hm. OK. The options for print. [Keep 
thinkinng aloud.] OK, I'm looking at the Options for Print, but 
I'm noot finding any help. I'm looking to print the record. 
Checf:xs under Using Print Records. OK, print records, you need 
to pnhnt selected, selected records default, we'll go ahead and 
print We want one copy, cover page, OK. Worried look at zero 
recorads formatted, but she waits. OK. 
QUESTIONHKAIRE COMMENTS: The system is very limited especially 
when someeone prefers to view previous search results and the 
system neeeris to go through the entire search process again. To me, 
this seems rather slow when an individual can search online in 
several mmnutes. 

Session 8 

A lot of search experience, has been on CD-ROM development 
teams before, but has never searched ERIC on the Mac. 
I Thousightful read of question and introductory screen. Asks if 
there is a native mode to interact with software or if find 
recorcuis dialog box is the only way, 

Um, Twhen I look at the question, I see two parts to it, and if I 
were going to start off before I even went online, I would want 
to, umn, conduct a reference interview because nutrition and 
exercrise is one idea and obesity and children is another idea, 
and r would want to know if the user or the patron was 
intereested in nutrition and exercise addressing the problem of 
obcsimy or if there are actually two different questions. 
Becaunsc nutrition and exercise and children is going to pull 
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down a lot of stuff on children's sports and sports programs in 
elementary schools, and obesity and children is going to pull 
down nutritional information and special problems of the 
obese, for instance, health problems. And so, there are two 
questions imbedded in the question as it's posed. Um, in the 
absence of a, uh, reference interview and just taking the 
question the way it's stated, uh, I think it can be addressed as 
searching for nutrition and exercise and with the age range as 
specified. Um, since I haven't used ERIC, at least the 
descriptors in ERIC for age ranges, I'm going to use the 
reference material to figure oujt if there's, um, an index 
category for that age range. I know that National Library' of 
Medicine Medline uses one, I don't know if ERIC uses one. 
Nutrition and exercise ought to be pretty obvious. 

TH Reads through intro to Thesaurus to get age level descriptor. 

OK, the, uh, indexing information tells me that they do have age 
level descriptors. Uh, and that the age level descriptor for 
children birth through eight years is young children. So it 
would have to be young children not, let's see, young children 
not infants. Because that would pull together pre-school 
children ages one through five and young children through 
eight years old, and eight years old is about third grade. So the 
age range statement is young children not infants, um. If, 
possibly, depending on the reference interview, rather than, 
um, age level descriptors, I might go to an educational level 
descriptor, uh, that would be primary education because that 
pulls together pre-school and kindergarten through grade 
three. So it would depend on if the bias ihat I got from the 
reference interview was, age was important or the educational 
level was important, uh, would be which descriptor I would 
choose to use. Since the point here is pulling as many 
references as possible I suppose it would be best to use both of 
them. So, young children not infants or primary education. 
That pulls together the age range. 

TH And, see what the thesaurus says about nutrition. OK, again 

we'd have to go back to the reference interview, because under 
the general heading of nutrition, there are, um, separate 
categories for breakfast programs, dietetics, eating habits, uh, 
nutritional instruction, obesity, uh, physical health, and so, if 
we use nutrition as a super term, uh, it will pull in the obesity 
arguments. And, let's see, exercise. OK, exercise is a heading, 
too. 
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E Now, let's see. I'm looking at the software to sec if there's a 
way to, um, access the inverted indices to see if this particular 
load of ERIC has hierarchical, or has the thesaurus terms 
underneath the subject heading terms, and I don't see anything 
except, um, search enter bar, and start search, and the Boolean 
and proximity operators. Let's see what help does for us. 

M Initial trouble getting to secondary menu in help, but he gets 
there. 

H Note that help is the only menu he looks under. Hm. I was 

looking through the fields to see if there's a way to expand the 
fields, and there doesn't appear to be, so, let's start searching 
and see what happens. [Tell me what you're typing in.] Um, 
just the first search term, young children not infants. 
Does click on NOT as opposed to typing. 

S- OK, the interaction might not be, uh, so good here because it 
looks like it's searching for single word postings rather than 
young children as a heading in and of itself, even though it's a 
heading in the thesaurus. Now I'm just waiting for the search 
to complete. And now I'm going to enter the second term that 
I decided on for the age range, and that is primary education, 
so add that using OR logic. Clicks OR, types primary education. 

Sf System adds #4 (previous search) to the beginning of this 
request. Now I'm just waiting for it to grind through the 
search. If the, uh, database isn't set up the way I expect it to 
be set up, in other words, if nutrition isn't a super term, and if, 
uh, exercise isn't a super term for all of the thesaurus terms 
listed under it, I would probably expand the search, and type 
in all of the terms that I had gotten from the reference 
interview plus anything else that looked like it might give 
likely retrieval. 

RT It's taking a long time to complete. If I had to do this a lot, I 
think I would be impatient with this application. Gee, I have 
time to get some coffee. I suppose I get spoiled with online 
systems. I can just type a whole logic in one line, and you don't 
have to worry about single postings. [Outside of this 
experiment, when would you consider breaking the search, or 
would you?] Um, in this particular case, I probably wouldn't 
break it, because of the fact that it's necessary to collect all the 
age range papers and then whittle it away with the application 
search terms. And so I probably would put up with having to 
wait so long for it. Usually I would break a search that was 
thrashing for a long time like this if it was becoming obvious 
that I had ill-selected search terms, that the retrieval was 
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going to be so massively large that I had picked the wrong 
thing, that I had just guessed wrong. In this particular case, 
the number of postings as it's rolling up there doesn't look like 
it's excessive and in a second I'm going to use AND logic to 
narrow, hut if it was blowing up, like hundreds of thousands of 
records I would break it off and try for something a little more 
surgical. Also, this is CD-ROM. This is here. I'm not paying 
online charges or anything so it can sit there and grind, and it 
costs the same to sit there and grind as not. Now if this was 
BRS or DIALOG or something, I'd have been gone a long time 
ago. 

? It seems to be done, but I don't have a cursor back yet. This is 
the experiment to see if "and nutrition and exercise" gets the 
cites I'm expecting. If the answer set is too large, I would 
consider adding "and obesity" to narrow it down to more of the 
exact parameters given in the original question. 

TH In the copious free time the search allows, I'm going to look at 
the index again and make sure I've done a good job selecting 
search terms. He means thesaurus. Since all of the terms that 
we've considered are narrower terms under the broader terms 
in the thesaurus, I am presuming that if there are records out 
there that talk about nutrition and exercise and either primary 
education or young children not infants that will hit on them. 
Now we won't have to go to the narrower terms because if it's 
set up the way the thesaurus says it is if you use a narrower 
term rather than a broader term, you won't get any hits 
anyway. If, on the other hand, it doesn't hit, we'll have to go to 
broader terms. I'll be prepared for that. OK, broader term for 
exercise is physical activities. Nutrition doesn't list a broader 
term. My retrieval seems kind of sparse. I think I'm going to 
try the broader term for exercise anyway. 

S4- Types in #7 instead of clicking. So I just type in, restate the 
search. Number 7 is the search for the age range, the search 
statement for age range, and nutrition and physical activities, 
which is the broader term, and see if we get, what kind of 
retrieval we get. And again, we have a period of leisure time 
here. Sigh. Can't tell if he is frustrated by slow search time or 
if he is just being a good volunteer. 

S- I'm speculating on the architecture of this particular product 
that rather than combining pointers from the inverted indices 
to determine record numbers, and then only going to the disc, 
actually searching the disc at display time, that this particular 
architecture scans the entire disc for every query. It's kind of 
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a goofy way to do it. It makes me reluctant to do the logical 
last step of this search which is to say, uh, search statement ten 
or search statement fifteen which offers insurance that in case 
somebody messed up at database construction time you don't 
lose any postings. Since physical activities is a broader term 
for exercise, there shouldn't be any in search statement fifteen 
that's not in search statement ten. On the other hand, since the 
database is put together by human beings, there just might be, 
and so, the best way to finish this search would be search 
statement 10 or search statement IS, but if this particular 
application is going to seek the entire disc again, um, to 
combine the 6 hits I got with search statement 10 with the 6 or 
8 that I presume Til get from 15, then it's a disincentive to use 
the product. Probably it's a disincentive to use it the way it's 
intended, OK, there's only 3 of them. Just combine line 10, the 
hits we got with exercise, and line 14 which is 3 we get from 
physical activities and nutrition, and hope that it will do this 
smoothly rather than have to flog the whole disc again. #70 OR 
#14. No, it's going to flog the whole disc again. Since I'm not 
going to do anything else, and since it's going to take a long 
time, can I go get a drink? Ill be right back. [Sure.] Break in 
tape. Note that the search is still continuing when he returns. 
RT If this application was a little bit faster and a little bit less 

awkward, I would probably be tempted to experiment around 
with some of the search terms a little bit more and see if there 
were words other than nutrition, for instance, that would get 
me where I wanted to go. Because the, uh, retrieval seems to 
be a little thin, I mean, there's somewhere between 6 and 9 
hits and to do the search properly you'd probably want to 
experiment around with some of the search terms a little bit 
more. On the other hand, realizing that it's going to take you 5 
to 10 minutes per experiment, or what I perceive as about 5 
minutes per experiment, it really doesn't encourage me to play 
around with the database at all. And I don't sense that, ERIC, I 
think, uses numeric codes sometimes ^i:?d online that's a very 
fast way to get at it, but if you have to seek the entire database 
going to codes rather than search terms, going to codes rather 
than search terms is not going to buy you anything. So it may 
be possible to do the search better in terms of increasing 
retrieval, but at the cost of spending a whole lot more time 
futzing with it. OK, looking at the question again, another way 
to approach it might be, rather than seizing on the terms 
nutrition and exercise, the, uh, another way, another question 
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expressed in the statement is^ what is the current thinking on 
teaching young children about nutrition and exercise. And 
maybe more citations would be retrieved if you used obesity 
and the idea of education or teaching. In a database like ERIC I 
think you'd be asking for it to blow up in your face if you just 
typed in education. Maybe there's a search term in the 
thesaurus for physical education. / can't believe he's still 
formulating more searches. Such a thorough searcher! 
Tenacious to get many hits even though system is so slow. 
What patience! No, that logic wouldn't get you any more hits 
than what you've already got because you'd be anding in terms 
and all that could do is possibly narrow your search. If the 
citation's going to talk about physical activity or exercise in any 
way, shape, or form not just in phys ed, and nutrition in any 
way, shape, or form rather thin just nutritional education, 
adding in terms for obesity and education is just going to 
narrow it, it's not going to make the, make the search any 
better as far as retrieval. It would make it better as far as 
precision, of course. But since the objective here is to get as 
many citations as possible, we're emphasizing retrieval. OK, 
there are 7 citations, and let's figure out how to print these. 
SHOW RECORDS FOUND- Ut's see. OK. That one's a good hit. 
That's a good hit. Yep, they're all good hits, so they're on point. 
Next Record click to go through hits, but a bit impatient waiting 
for clock to return to arrow. Let's see, how do we print these 
guys? [You can just print the first five.] Print records. OK, Til 
save search history. Let's see. Hm. Chooses Select Fields and 
changes to all. The, um, option for selected records is not 
selectable. Ah. [Ah, what?] Ah, we have to go into retrieved 
records and print which ones we want to see. 
Clicks ? and does move the ? to All Records button. Select it by 
clicking on the record number. OK. So, we'll look at the first 
five. When paging through, waits patiently for clock. I'm just 
paging through and selecting the first five so it will print them. 
Wait a minute, there you go. 

Now, go back to print records. Search from reco J. Trouble 
remembering where he'd found print command. Select print 
records. Selected records, OK. I sent the first five records over 
to the laser printer. [OK, are you ready to stop?] After search I 
asked him to go back to the search history window. With only 
a little hesitation he clicked the close window box and got back. 
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QUESTIONNAIRE COMMENTS: 1. System is way too slow. This can 
discourage searching by experiment. 2. No "native" or "online" mode 
for experienced searchers. 

Session 9 

CR Glanced at quick reference, 
UM flipped through manual, 

TH but checks thesaurus for terms. Looking for help on concepts. 
OK, the first thing I want to do is to become familiar with the 
information that you've provided here to help me with my 
search and determine what concepts or terms I should use in 
my search, because I'm not as familiar with the Mac version of 
this database. Is it similar to using? Can I ask a question? [I 
may or may not answer it.] I'm just saying, is it similar to the 
other? [The CD-ROM is the exact same thing.] So that the 
search, to search it and everything else is the same? [You may 
find some differences.] OK. OK, I'm using the thesaurus right 
now, and I'm looking up the term obesity. OK, it's not here. 
Now, I'm looking up, I'm trying to determine what, OK, the 
current thinking on teaching young children, preschool and 
kindergarten through three. Whether that's primary school or 
the term for that group would be. OK, I found preschool 
children, age two to five. Primary education, OK. So I'm going 
to try the term, um, let's see, preschool and primary and 
nutrition and exercise. And then primary school. I'm allowed 
to do a number of searches, right? OK. Primary education and 
nutrition and exercise. [Tell me what you're typing in.] OK. 
"Preschool and nutrition", I'm checking here to see if I spelled 
nutrition right. And exercise. 

Sf Types and vs. clicking it. And just enter it? Oh, OK. 

Sf Enter vs. Start Search. OK, they're showing the records here for 
each of my terms. And I've found two records. OK, now, I'm 
going to look at those records. No trouble with mouse 
movement. Health practices reported by daycare parents, 
implications for early childhood educators. Well, that doesn't 
address the issue of obesity. Let's see what the next record is. 
OK, now here's one that seems to address. So, do I print it? Is 
that what the instruction says? [I'd like you to print at least 
five.] OK, OK. As I choose? OK. Goes immediately to file menu 
to print. 

E Selected begin print from record #2. In print dialog box, said 
to print from page 2 to page 2. Is there any particular way you 
want it? Separate page? [It's up to you.] [Tell me what you're 
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typing in.] OK, what I'm printing? Ym printing an article called 
ready, set, grow, health education for 3-5 year olds» 

Nf Used down arrow to look at abstract, OK, the abstract tells me 
it's intended for use in family,daycare, preschool centers, 
professional corporations, institutions, and in home. It's a 
comprehensive health education curriculum for three through 
five year old children, and contains units designed to 
sequentially teach concepts about health, mental health, family 
living and safety. Also deals with nutrition and exercise. OK, 
I'm going to do another search. [Keep thinking aloud.] 

NE OK, I'm trying to figure out how I start another search. OK, so 
I'm thinking here, should I get help? 

Q Clicked ? to get help on getting back, but re-clicked to get 
arrow. Checked under File, Edit and Help, then decided to 
restart session. If I went restart session, does that get me back 
to the search, I'm thinking. Now I'm waiting for it to do 
something. OK, I'm back to ihat. OK, I'm going to enter another 
search now, and this one's going to be primary education and 
nutrition and exercise. So far I'm not coming up with any 
records for this search. Search at 8% complete, I should know 
better than to use education in ERIC. I'm still not getting any 
records on this search. Search 66% complete, OK, it came up 
with 2. I'm going to view those now. OK, Tve come up with an 
item called health curriculum guide, grade 1. OK, and the 
abstract tells me that this is a curriculum guide, supplements 
the Alabama health education course of study which offers a 
comprehensive plan, sequential curriculum for grades K-12. 
OK, I'm going to print this record. 

E Again prints from page 1 to I, did seem to notice that 2 records 
formatted, but doesn't comment on it, OK, now I want to view 
the next record that this search provided. OK, the title of this 
one is project hardy heart, a cardiovascular fitness curriculum 
for first grade children. Now I'm going to view the abstract. 
This physical education and health education curriculum guide 
specializes in the healthy heart. Is designed for use by both 
classroom teachers and physical education teachers. Part one 
outlines the physical education curriculum for children in first 
grade. Included are a variety of physical education program 
activities which focus on improving cardiovascular endurance, 
enhancing basic motor skill development, increasing levels of 
strength, flexibility, and balance. It does have an emphasis on 
exercise and nutrition. I'm going to print this one, too* 
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E Uses same strategy to print this record. OK, I need to formulate 
another search here, so I can get as many records as possible 
on this topic. I think what Fm going to use is a broad term 
which is early childhood education and these two other terms, 
nutrition and exercise. Let's see. 

^E Checks under File and Edit, then Restarts session again. This 

time Fm going to use and nutrition or exercise. To broaden my 
search. OK, Fm entering the search here. 

L Early childhood education and nutrition or exercise. This is 
interpreted as (early childhood education and nutrition) or 
exercise. But window only shows to word nutrition, not the 
close parentheses. This isn't really the logic she wants. OK, so 
far this hasn't found any records. 9% complete, 0 records 
found. [If you want to try to print just one more I think that 
would be fine.] OK. 

TH Appears to wait patiently during search, looks through 

thesaurus to formulate next search. While this is doing that, 
my, if we were having a reference interview, one of the 
questions that I would ask is that are yon also looking for 
material on theory. And so, Fm wondering if a search on that 
is necessary, in that area. OK, it's come up with 2111 records. 
Sigh. OK, the final results are 3113 records. Fm going to 
narrow this search a bit and do early childhood education and 
nutrition and exercise. Fm going to now formulate a search for 
theory. First I think health education. OK, Fm finding health 
education, nutrition instruction is a term that might apply here. 
Now Fm looking up theory. OK, um, I believe my search terms 
are going to be nutrition instruction, um, early childhood 
education, nutrition instruction, and nutrition instruction. And 
theories. So . far my last search hasn't found any records. OK, 
one just appeared. OK, 2 records were found. Now Fm going to 
view those records. OK, the title of the first item is health 
update, preventing heart disease begins in childhood. The 
abstract tells me that this article maintains that the prevention 
of heart disease begins in childhood with good dietary habits 
and healthcare. Provides parents and childcare professionals 
with guidelines they can use to help prevent heart disease in 
children. Well, I think I'm going to look at the next record 
here. Family health is the title, and it's bilingual. These are 
booklets on family and child health. This is part of a series of 
22 booklets specifically designed to help parents understand 
their children and help them to learn the effects of stress on 
family life, covers issues such as the causes and effects of 
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stress. OK, this doesn't seem to apply to what I'm looking for in 
terms of classroom education. OK, I'm not going to print either 
one of these. 

NE Oh, did I quit? [Just click on MacSPIRS.] Here? Oh, OK. Quit 
instead of Restart Sessit i, by accident? Hoping that was way 
to get back? OK, I'm going to enter another search here then. 
Early childhood education and nutrition instruction and 
theories. OK, I'm wondering if I should truncate theory. 

QR. So, I'm looking at the guide to find out about truncation. OK. 
So, I'm going to truncate theory. Using a star. OK, while this is 
going on, I'm going to formulate another theory, that is, another 
search. And that would be preschool, um, primary and 
preschool and preschool and nutrition instruction. I'm going to 
change this to primary or preschool. Change made on paper to 
search terms just stated. Still searching early childhood 
education and nutrition instruction and theories. I'm debating 
here about whether to make it primary education or leave it 
primary in this next search. 

TH So I'm going to look up in here primary and see if I'd be pulling 
up a whole lot of things. Primary education. So far I'm not 
coming up with any records on this search. 79% complete, 0 
records. OK, there were no records found for theories. So I'm 
going to do one more search here. And I'm going to use 
primary education or preschool and nutrition instruction. OK, 
we found one record. [If you want to print one of these when 
you look at them and then stop that's fine.] OK. OK, we found 
four records. This search has produced 19 records so far. / 
think she's afraid that Break Search will lose records found to 
this point. I told her to print one, yet she waits patiently for 
search to complete. 62 records were found using that search. 
Now I'm going to view these records. OK, this one, the first one 
is entitled education for child survival and development in 
Africa. This is a UN document. I don't think this would be 
appropriate for what we're looking at unless this person has an 
international focus. And obviously, well not obviously, but it 
wouldn't apply to obesity either. The second one . entitled 
effects of parent augmentation of preschool curriculum in 
children's knowledge scores. Compared gain scores of one, a 
group of middle-class children in daycare centers whose 
parents augment the daycare curriculum at home, two, a group 
who receives only the daycare curriculum. I don't think this so 
much applies. Special cooking friend is the second, I mean, the 
name of the third. Add spice to head start nutrition program. 
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Describes American Home Economics Association's volunteer 
nutrition consultant project in Which home economists teach, 
advise, and share their nutrition expertise with Headstart 
children and families. [Why don't you just print that one?] OK. 
[I don't mean to rush you, but I think you've done a good job.] 
OK. 

E Checks options for print under file menu, still looking discretely 

for way to narrow printing, and uses same strategy as before.. 
QUESTIONNAIRE COMMENTS: None 

Session 10 

Unfamiliar with the Mac, but he searches for a living. Windows 
experience. 

Concerns about obesity in young people, current thinking on 
teaching young children, preschool, kindergarten through third 
grade about nutrition and exercise. Seems like a fairly well- 
defined search question, which is good because this is not 
interactive. I couldn't seek further clarification on the 
question. So, I'll start by familiarizing myself with the 
database itself. OK. 

Sf I'm not familiar with the Macintosh, I haven't used it before, 
however, I've used Windows for IBM PCs which is a similar 
interface, at least it looks it from the outside. 

I I'm just going to read through the introductory information. 
OK, I've used the ERIC database before for work I've done for 
library research as well as professional research. OK, enter the 
search. I think I'll begin with obesity. I guess my initial 
question would be if I'm able to refine the searches once I 
begin them, and I'll find out soon enough. Start search click. 
Search is in process, and I spelled obesity wrong. I'll try that 
again. OK, there are 228 records. 

Sf It looks like I can combine the search number and then use 
Boolean operators to refine the search. I'll try that. [Tell me 
what you're typing in.] I'm typing in search number 2 which is 
obesity and I'm using the Boolean operator and, anding in 
preschool. #2 and -- types in and. A question that comes to 
mind is if I can use parenthetical refinement I could put, for 
instance, number 2 and preschool, if I could open a parenthesis 
and put preschool or kindergarten, that way it would search for 
number 2 and preschool or kindergarten. I don't know if I can 
do that. I'm just going to search number 2 and preschool, with 
preschool being all one word. OK, it looks like the Boolean 
operator works as I anticipated, and I have a total of four 
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records showing. The search question asks for, in addition to 
preschool, kindergarten through third grade. I'll go ahead and 
try that. I'm searching for obesity, which is search #2 and 
kindergarten. See what I come up with there. OK, came up 
with 2 hits. The second part of the question has to do with 
nutrition and exercise, which may or may not have to do with 
obesity. I think at this point I'm going to look at a couple of 
records and see what I'm pulling up. There's also the concept 
of currency meaning I assume in the last, it's difficult what you 
mean by current. If I were speaking with someone I would 
ask them to further define currency because that could be in 
the last ten years or the last year. So, that's a question that I 
have. Experienced searcher looks at currency of info. I'm 
going to take it from the last three years. Now, in order to do 
that I've got to know how to limit by year. 

Q I don't know how to do that, but I see a question mark. I'm 
going to check that out. I'm pushing the question mark, uh, I 
assume that would be help, but apparently it's not. 

E Fails to deselect ? icon. 

H I'm going to go up to the help screen, to the help pull-down 
menu. I'm going to the fmding records, narrowing a search. 
OK. No problem using secondary menus. I'm apparently in the 
help menu concerning help because I clicked the question 
mark. Now I'm going to do it without the question mark and 
try that. OK. Looking for help on fmding records, messed that 
up again, narrowing a search. What I'm looking for is how to 
limit by date. OK, first it tells me about Boolean operators or 
you can search in a particular field, limit fields, language, 
publication date. OK, that's what I'm looking for. Most 
databases have specific limit fields, py for publication year, I'm 
reading from the menu, that you can use with other search 
terms to focus the search. OK. I don't see a, um, list of those 
fields, OK, now I'm to, in the help menu, it says searching limit 
fields. Use in or = to search in this field, so if I wanted to 
search for publication year I use the prefix py, I'm looking, I 
want a date range, I think this goes only through 1991, so I'm 
probably looking for articles from 1989 to 1991, and I'm 
looking for how to do a date range. I found it, using a hyphen 
to do date range. 

Nf OK, I'm going to exit out of the help menu, I'm going to click a 
box in the upper left comer which worked, so. OK. Now, going 
back to my original search, I have 4, possibly 4, or 6 applicable 
records. I'm going to combine those at this point. #4 or search 
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#6, that gives me both the articles that had to do with obesity 
and preschool, and the articles that had to do with obesity and 
kindergarten combined into one set which gives me six records 
as expected. I'm still concerned about the nutrition and 
exercise aspect, and I'm also concerned about the first grade, 
second grade, third grade. Because I've got preschool and 
kindergarten and I've got second through third grade yet to 
look for. For the time being, I'm going to search #7, which is 
kindergarteu and preschool and obesity, and publication year 
equals, let's go 1988 to 1991. See what we come up with. 
Search is in process. Um, but I don't come up with any records 
here. I'll have to approach it from a different way. Or, I could 
re-interpret current to be the last ten years. Um, I'm just 
waiting for the search to complete. 
RT It's a fairly complicated search as opposed to the searches I 
was using before ^-bviously it's taking a longer time. I'm 
accustomed to using online services which operate faster, 
which leads to a little frustration when you have to wait on the 
CD-ROM to look through, but at the same time I realize that it's 
not costing me money, so which has sort of led to the 
roundabout approach to this search. I guess in going in to this 
search, while I'm killing time, I, my objective was not only to 
answer the research question, but to familiarize myself with 
the source. If I were paying for it I would have planned the 
search ahead of time. I also would have spent some time 
looking through the hardcopy manuals. OK, zero. Search is 99% 
complete, and it's still looking for records. That's kind of 
strange. So anyway, I didn't use any of the hardcopy 
information before using. OK, we've come up with 2 articles. 
I'm going to look at these articles, see if they are close to what 
we're looking for. Finds Show Records Found button without 
problem. 

Nf OK, I'm using the arrows on ihe side of the document to look 
down through. OK, this first one has to do with preschool, um, 
and the second has to do with preschool nutritional, let's see, 
OK, I'm actually reading through the abstract to see, I feel that 
the first record I pulled is applicable. Um, looking at the 
second record, and I'm looking back at the search question. I'm 
not sure about the applicability of the second. It seems to 
focus on low-income, specifically, and I'm not getting the 
impression from the search question that that would 
necessarily be applicable. It's possible that it would be helpful. 
But, it's not an ideal record. I also noticed that, OK, the records 
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are 1988 and 89 which is a little early, again it depends on how 
you deHne current. Sigh, well what I'd like to do at this point, 
is print out the record that I feel is applicable, and just get it 
out of the way. Um, I'm not sure that I would do this, and it 
says to print the first five records. It didn't ask where the 
record will be printed out, but I imagine it will be printed 
somewhere. Um, OK. So if I wish to print this record, there's 
an option for printing records. OK, search number 7, printing 
record 1 . 

OK, I don't understand print records. Cancel and go back. First 
record, file, print records, OK, I'd like to just select one record 
to print. 

Tries to click selected records, but not highlighted. I 
accidentally picked choose fields, I didn't want to do that. So, 
I'm going to cancel. Um, I don't want to print all records, I just 
want to print record 1 of 2 of search number 7. I don't see 
how to do that. It looks like it's going to print records 1 and 2 
because it says all records. Oh well, I'm not going to get hung 
up on that, I'll just print that out. OK, I'm assuming that there's 
a printer hooked up on the other side of the lab somewhere. 
[Right.] OK, um, what I'd do at this point, I can't figure out how 
to print the one record, and I don't feel like spending any more 
time on this, so I'm just going to print both and manually 
separate the two. Now it's in the process of printing. OK. OK, 
so I feel like I've received one record that was of use. Now I 
need to go back in, and I need to find something that has to do 
with, actually I think I'm going to redo search #7. I'm , redo 
search #8, which means I'm going to us6 kindergarten, 
preschool, obesity. I'm going to go back from 1982 to 88 and 
see what kinds of records of records are available, just to see 
what was going up there, preschool and kindergarten. OK. 
Easily closes window of retrieved records window to get to 
search history. OK. I'm typing in search #7 and 1982 to 87 
just to see what kind of records we come up with there. And 
again, it took a long time last time, it's going to take a long time 
this time. Sigh. OK. If I wanted to do this quicker, range is too 
wide, please use a narrower range. Evidently, the software 
can't handle that, so, I'll use from 1985 to 87. 
Range error message. Why? It accepted PY-1988-1991. 
I went in to edit the search, which is sort of a convenient thing, 
something that you can't do if you're working online. No 
problem correcting search. OK, I'm going to go from 84 to 87. 
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At this point I'm aware of the length of time that it's taking to 
do this. 

? If I wanted to, uh, I guess if I were doing this, knowing how it 
works now, again the range is too wide, which, I'm not certain 
I'm understanding what that means. I'm assuming it means 
that the range of dates is too wide, so let's just do 1986 to 87 
and see if that works. Maybe I'm not understanding what that 
means. OK, um, looks like it's not working. Let's just try one 
year. PY=1986. OK, let's just try 1987. That didn't work. 
There's an annoying black space, and I don't understand what 
that is. OK. Um, try the search again. Apparently it could 
handle that. OK, well that's a limitation, as far as I'm 
concerned, not being able to search the date range. Um, sigh, 
OK. So, let's go for a different approach. Again, I'm concerned 
about first through third grade, so I'm going to type in first 
grade and number 2. Looking for obesity and first grade. Um, 
I wonder if there's a term that covers first through third grade, 
like elementary education or something like that. I guess I 
could look in the thesaurus while, I'm looking to see if there's 
any reference to what kind of term I should use to cover first 
grade, perhaps I'll also look, um, at elementary. Again if I 
were doing this search and being charged for it, I probably 
would have looked through the thesaurus first or used an 
online thesaurus. That's actually something that, something 
else that I could have done, is checked the online thesaurus, 
and I'm not sure if there is one. Online thesaurus meaning 
terms that are indexed. 

TH Uh, I haven't seen any mention of that, I think I'll just continue 
to look through the hardcopy. Starts to look for online 
thesaurus. Moves cursor to file and edit menus, but then stops 
and goes back to hardcopy. Um, OK, I'm going to look up 
elementary. The, um, elementary school children, elementary 
education, education provided at kindergarten, grade one 
through six, seven or eight. OK, problem with that is that it 
includes grades five, six, and seven, which is not listed among 
the, they only want kindergarten through first grade, so. 
Elementary education apparently would not work, however, 
the related terms does list grades one through three, so I'll use 
that. I'm looking for grade one and obesity. Um, again if I 
were starting the search over again, or if I'd used this before, I 
probably would have included all of the terms that I could 
think of, andi uh, in the first search, and then limit it down 
from there. That would have significantly reduced the amount 



122 



114 

of time it takes, um, but since I didn*t know how the system 
. worked, aad how it handled Boolean operators, then, uh, Fm 
using this to learn. Presumably if I were going to use this 
system again, then the next time I could come in, without 
having to deal with the lack of knowledge of the interface, and, 
it does seem fairly straight forward, and if you're just someone 
who doesn/t necessarily search a lot, uh, I think the fact that I 
have searched before perhaps led me to be overly analytical 
about the whole thing. Which has caused me to take more time 
than necessary. I, uh, couldn't find anything in grade one and 
obesity. 

Sf Also uses return key to start search. I'm also trying grade two 
and obesity. It's doing the search. If it's some, I guess, 
question I would have is, if it's a study, the studies themselves 
that include grades one through three might not be limited to 
grades one through three. And it depends on how the ERIC 
thesaurus is put together whether if it includes all the grades, 
if they separate the grades out, for instance, if it's a study that 
includes grades one through six, it, they also indexed, the 
thesaurus indexed the descriptors by their individual grades 
also. That's just something I don't know. I assume that they 
may use the broader term elementary education or something 
like that, and, uh. OK, we're also not coming up with anything 
for grades two and number two. It*s just a dead end. I'll try 
grade three and number two. 

Sf OK, um, it looks like an interesting aspect of this is the search 
history, uh, it looks like I can go back and see what I've done 
before. At this point I'm getting a large number of searches 
which is, which is really too many to keep track of in my mind. 
Uh, again, I attribute that to the fact that I haven't used this 
before. And, in addition to, trying to come up with the answer 
to the question, I'm also trying to figure out how I would 
handle my approach to the issue. OK. I'm going to go back, and 
I'm going to find, uh, OK, preschool, kindergarten, grade one, 
grade two. and grade three. I'm going to do a comprehensive 
search, just to include all of those, then I'm going to and in 
nutrition or exercise, then Fm going to print some records and 
be done with this. Or that's the plan anyway. Kindergarten or, 
um, preschool or grade one or grade two or grade three. When 
combining searches, he stops at the end of the line instead of 
continuing to type and letting window move. 

RT Again we run up against the fact that it's a CD-ROM so it takes a 
long time when you complicate the search. When you're used 
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to fairly instantaneous results it gets annoying. Especially with 
a small set of records. ERIC 1982 to 1991, I guess that's 
actually, there's a large number of records there, but I've 
searched much larger collections of records much faster and, it 
gets annoying. Tm also aware that ERIC's available online, free 
places, which would take less time, and give you a complete set 
up-to-date. This is September 1991 which, at this point, is a 
year old, and so Fd have to question the, uh, the currency of 
the results that I get. OK, um, ziUioifft of records, uh, actually 
four thousand. And it's lumbering along. 
Let's see, interface, I think is, um, it's good that it's telling me 
what percentage it's finished with the search, lessens the 
frustration a little bit. [At what point would you consider 
breaking the search?] Um, I would probably, this is obviously 
an unwieldy amount of records, but I would like to, if I have all 
the time that I need, which I do, then there*s really no need to 
break it. I don't know how much nutrition is going to, what I 
can do at this point is, even if I have ten thousand records or 
fifteen thousand whatever, I can immediately use 1991 which 
is going to limit it tremendously, and I can still include all, all 
the records. By breaking the search, I don't know at what 
point it's broken the search. For instance, if I knew that it 
looked at the most recent records first, um, then Fd break the 
search now, but it's over halfway done. To limit it right off, if I 
knew what the definition of current was then I could have 
included that. I could have anded in py=1990, 1989 to 1991, 
but then I might run into the fact that it can't handle that date 
range like it told me before which was frustrating. Sigh. So, it 
seems like a long time, but in reality this whole search is going 
to take less than two minutes. If I were going to do this by 
hand, of course, it would take weeks. So, you just have to 
temper the fni«:tration with that. If I were concerned about 
time, of course, there are a couple of other ands that I could, 
that I could and in, I could and in nutrition, I could and in 
exercise. OK, let's see. 12,000 records, Fm going to take #31, 
I'm going to and in py=1991. Complex search request, try 
using clear command in search menu first or simplify search. 
OK, now that makes me angry. 

Um, I don't know what it means, and I've just spent a lot of 
time waiting for it to trundle through 12,000 records. And I 
don*t think it's a complicated a search. And Fm not exactly 
sure what it means by simplifying search. I wonder if I go 
back in and enter those, all those, if it's going to tell me again 



that it's complex after I wait ten minutes for it to go through. 
Complex search request. Since it's giving me this message, I 
also wonder if it's actually giving me all the records that are 
available in kindergarten, preschool, grade one, grade two, and 
grade three. Um, [indecipherable] I'll try nutrition. Complex 
search request. Huh. OK. One, two, three, four, five. Five 
search terms really isn't that complex. Scrolls left to right to 
count search terms. So, nutrition. We'll just go at it from 
another angle. Nutrition and py=1989. We'll go back to 1988. 
1988, 1991. Sigh. 

Range is too wide, please use a narrower range. [Try py 
greater than 1988 or greater than 1987 and see what it does.] 
OK. PY greater than 1988. Like that? Ah, another thing that 
might be, OK, that's working. I'm sure I saw that range, and 
I'm sure I saw that on the help menu. [Well, you used it once.] 
Yeah. [And it worked.] Right, I think possibly the search 
history has to be stored somewhere, and possibly the fact that 
I'm up to 32 searches, it's getting annoyed. Not that I would 
ever put human traits on a stupid computer. [Right, right.] OK, 
it's finding some records on nutrition. I guess it's good that 
I've done this. I'm learning the limitations of the source. 
Knowing the limitations I probably wouldn't come back and use 
it again because, um, because I don't exactly know at what 
point it's going to decide not to respond, it seems to be sporadic 
on what's acceptable and what's not acceptable for search 
terms and for date ranges and for that sort of thing. If I'd paid 
a lot of money for it, I'd probably send it back. Um, the 
question I always have in doing these searches and receiving 
errors is asking if I've done something wrong. Um, in this case, 
I don't think that I have. But there is some sort of limitation 
on what you can use on date ranges and number of records 
and, and that I think should be spelled out, 'especially in the, 
uh, advertising literature. For instance, I've never had DIALOG 
tell me that my search strategy was too complicated. I have 
had other systems tell me that, but ERIC's available on DIALOG. 
[You're just a complicated kind of guy.] I suppose. I do admit 
that my approach is a roundabout way. Maybe I was trying, 
deep down, to see if I could screw up the computer. [And I 
appreciate that.] I think the, um, desire to break the search 
increases as the length of time you spend sitting in front of the 
computer increases. For instance, if I were waiting for this 99% 
complete to finish up a few minutes ago then I would probably 
be much more relaxed than I am right now. What I would 
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want to do if I were satisfied with the searching capabilities, 
I'd want to go into some of the options at the top of the screen, 
like the show options and there were several other search 
options and that would be interesting. I think it's positive that 
this is available on Macintosh because Macintosh has a 
reputation fqr friendliness. Urn, I'm not sure if that's true or 
not, but Apple has worked awful hard to create that, and that 
means the whole idea of information being accessible to 
everybody is that much closer. And, most people aren't going 
to have this complicated of a search. So, probably for, I don't 
know, 70, 80% of the people who would use this, the, um, they 
wouldn't run into any problem. I couldn't imagine most people 
ever piling up 33 searches. OK. Now I'm mad. 
Error message after 99% complete for some time. The range is 
too wide, please use a narrower range. I'm angry because it 
told me that after it had searched for five minutes. So, the 
assumption is, I've got this client that's waiting on me to 
answer this question, and I've been messing around with this 
for half an hour, and I can't go to the client and, say, look, I 
don't like this. So, um. Sigh. I'm looking through, I'm right 
now contemplating clearing the searches and starting from the 
top. I assume that the reason it's being so annoying is because 
I've gotten so many searches. This time I'm going to start with 
a comprehensive search, limit it by the date, and, uh, print 
some stuff out. That was the idea I had before, wasn't it? He 
does clear searches from search menu, but is so frustrated he 
doesn't investigate other options. Preschool or kindergarten. 
[Forget the date.] OK. [I'll make it easier for you.] Just get the 
heck out of here. [I don't want you to throw anything through 
the screen.] I wonder how, if other people ran into these 
limitations. That would be my question. Or if it's just me. Or 
do computers just hate me. [Tell me what you're typing in.] 
I'm typing in the, the grand search. The search that retrieved 
12,000 articles, or 12,000 hits. The one that I should have 
started out with to begin with if I knew the system. Preschool 
or kindergarten or grade One, oops, I did. that wrong. It's a 
good thing I read that back. Kindergarten or grade one or 
grade two or grade three. [Check your one over here.] I think 
my search efficiency goes down after a certain time. OK. 
Preschool or kindergarten or grade one or grade two or grade 
three. 

OK, this time it tells me it's a complex search request from the 
get go. It's odd that it did the search before. It obviously did it 



and stopped and didn't give me the complete. It's bad that it 
did that before because I could have assumed that it completed 
the search as it said it did, and, uh, that would have just been 
wrong. It looked like a very low number, but it didn't tell me 
that it was too complex of a search before. Um, sigh. Let's see. 
Erase. Let's, uh, limit it to, simplify the search. Again, I don't 
think any person that docs searching would say that this is a 
complex search. I know, I just don't think that that qualifies as 
a complex search. And now, I don't have the excuse of having 
33 searches done before. It's just a lousy product. Or, py, OK, 
I'm going to do preschool or kindergarten and nutrition. We 
already did obesity. [Print a couple of these records and I'll let 
you go free.] Thanks. So, while we're waiting for this search to 
complete, I'm not satisfied with the product, I'm not satisfied 
with the messages I've received, um, I think that to be an 
effective product it would have to have a more flexible 
searching, uh. Actually the searching strategies are available, 
it's just that it can't do them, so, um. Again, I just tried to or * 
five terms together that were found in the thesaurus and uh, 
and, I don't consider that complex, and I don't think I'm wrong. 
I'm going to print out, I'm just going to print out the first four 
from the file. Print records. Now this is what I didn't 
understand before. I want to select the records that I want to 
print. I don't want to print out all 191 records. It says starting 
printing from record number. 

E Well, I'm going to start from 185 or 186 so it will print out 186 
through 191. I would like to select records, I don't understand. 

Q Let's look at the question mark. Um, prints a record retrieved 
by the most recent search, prints all records by default if you 
have not selected a subset of retrieved records by clicking their 
identifying numbers in the retrieved records window. By 
clicking their identifying numbers in the retrieved records 
window. Does click? and moves it to Selected Records, but 
doesn't understand. 

? I don't know where the retrieved records window is. Hm, that 
confuses me. I'm not going to stick around long enough to 
figure it out. I would have rather had needles shoved into me. 
[OK, OK, well if you'd rather have that, I'll go to plan B.] Let's 
see, one copy, pages, and if it starts printing out more than this 
then we'll shut off the stupid printer. [OK, I think that's fine.] 

QUESTIONNAIRE COMMENTS: Frustrated with product. Limitations 

became evident when multiple search terms were used. Interface 
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was friendly but misleading; I felt like it didn't do what it was 
supposed to do. 
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APPENDIX B 
SCRIPT 



The following script was read to all subjects prior to beginning the 
search on ERIC: 

This information is being read to you so that each participant 
will receive the same instructions. You will be given a topic for a 
search on the ERIC CD-ROM on this Macintosh [point to Macintosh]. I 
have placed several resources here that you may want to use, but 
you are not required to use them [point out the ERIC Thesaurus and 
help manuals]. As you search, I would like for you to think-aloud. 
Tell me what you are thinking, questions that pop into your mind, 
problems you might foresee, what you are typing in, etc. Try to 
voice the logic you are using. Your spec h will be recorded by this 
tape recorder [point to recorder]. Also, I will be observing you from 
this chair [point to chair], and I will be making notes on my 
observation. 

At the conclusion of your search, I will give you a 
questionnaire to complete. You have the option of taking it on paper 
or on the Macintosh. You can let me How at that time which is more 
comfortable for you. 

Try to be as relaxed as possible. This is merely an observation. 
You are in no way being graded or judged on the outcome or 
timeliness of your search. I assure you of strict confidentiality. The 
data collected here will not be able to be traced back to you. Should 
you feel uncomfortable at any time during the search or during the 
questionnaire, you may stop. 

Please feel free to ask any questions you might have before 
you begin your search. Once your search has begun, I will discourage 
questions. After your search is completed, you may again ask any 
questions. Do you have any questions? 
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APPENDIX C 
SITE SOFTWARE LICENSE AGREEMENT FOR QUIS 



The following six pages are a copy of the signed Site Software License 
Agreement. This agreement is between the Human/Computer 
Interaction Laboratory at the University of Maryland at College Park 
and the author for the period of 12 months. 
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UNIVERSITY OF MARiXAND AT COLLEGE PARK 

OFHCE OF TECHNOLOGY UAISON • GRADUATE STUDIES AND RESEARCH 

UNIVERSTTY OF MAR YLAND AT COLLEGE PARK 
SITE SOFTWARE LICENSE AGREEMENT 



This License is made and entered into by and between the University of Maryland at College Park 
(hereinafter UMCP), and the LICENSEE, executing this License by a duly authorized representative. 

1. DEFINTTIONS 

L 1 "Computer Output" means the paper version or output of data of the Program and any modification 
or update of said Computer Output delivered to LICENSEE. 

1.2 "Program" means the computer software object code version of Questionnaire for User Interaction 
Satisfaction (QUIS) and any modification or update of said software delivered to LICENSEE. 

1.3 "Designated Computers" means tiie single central processing unit ("CPU**) and its associated units 
as identified in Schedule A. 

1.4 "Designated Site" means the location of the Designated Computer as identified in Schedule A. A 
"Designated Site" consists of a building, or building and immediately adjacent buildings, all of 
which are addressable by only one single postal address. 

1.5 "LICENSEE" means die party identified as such in the signature block of this Agreement 

1.6 "Licensed Materials" means the "Computer Output" and "Program" as defined above and any 
documentation, manuals, textual material, or otiier information provided by UMCP to LICENSEE 
("Documentation"), 

1.7 "Use," "to Use," or "Using" means utilizing the Licensed Materials in the course of the operation 
of the Designated Computers. 

2. GRANT OF LICENSE 

In consideration of tiic payment of the fees and charges set forth herein and tiie obligations undertaken by 
LICENSEE, UMCP grants LICENSEE, and LICENSEE accepts, a nonexclusive, nontransferable license 
to the Licensed Materials. 

3. SCOPE AND LIMITATION OF RIGHTS 

3. 1 The License granted in Section 2 authorizes LICENSEE to: 
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(a) Make copies of the Computer Output for use at the Designated Site ocniy. 

(b) Install the Program on the Designated Computers located at the Dcsicmatcd Site- 

(c) Use the Program only on the Designated Computers. If the Prognram is to be used at a 
location other than the Designated Site, an additional license will be rrequired for each site. 
A license fee for additional sites will be charged to LICENSEE, No service bureau or time 
sharing use of the Program is authorized by this Agreement 

(d) . Make copies of the Program in object code (machine-readable) form -which includes a right 

of installation and use in accordance with this Section and make ccpmes of the Documenta- 
tion for LICENSEE'S use in conjunption witii the authorized use of ome Program. 

(e) LICENSEE shall maintain records of the number and location of all c copies of die Licensed 
Materials and shall send these records to UMCP as copies are made-. The original and any 
copies of the Licensed Materials, in whole or in part, which arc mace r hereunder shall be the 
property of UMCP. 

3.2 Any copies, adaptations, transcriptions or merged portions of the Licensed NMaterials, including all 
portions thereof, shall include all proprietary and restrictive legends and noocccs incorporated in the 
copy furnished to LICENSEE or furnished to LICENSEE by UMCP from mme to time. 

3.3 Copies of the Computer Output or portions tiiereof are not to be included : in research reports or 
other publications resulting from its use without prior written authorization zvy UMCP. 

4. FFRS AND CHARGES 

4. 1 LICENSEE shall pay UMCP a license fee of One Hundred Dollars (SICC.OO), made payable to 
"Office of Technology Liaison," due and payable at the time of LICENSEEE's execution of tiiis 
Agreement. Payment and LICENSEE'S executed copy of this Agreement snaould be sent to: 

Director 

Office of Technology Liaison 
4312 Knox Road 
University of Maryland 
CoUege Park, MD 20742 

4.2 LICENSEE is solely responsible for and shall, if requested, reimburse U MCP for any taxes 
(including sales or use taxes and intangible taxes) resulting from LICENSEEE s acceptance of this 
License and LICENSEE'S possession and use of the Licensed Materials. L"?>MCP reserves the right 
to have LICENSEE pay any such taxes as they fall due to UMCP for remiisance to the appropriate 
autiiority. 

5. TFRM AND TERMINATION 

5. 1 The term of tliis Agreement shall be twelve (12) montiis, effective from ice date of signing by a 
duly authorized agent of UMCP and shall apply to the Licensed Materials any future modifica- 
tions thereof delivered by UMCP to LICENSEE. LICENSEE'S accqptancre of any modification 
will be conclusive evidence of LICENSEE'S agreement that the use of such modification is 
governed by this Agreement 
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5.2 This Agreement shall automatically terminate if Licensee uses, distributes, allows access to, 
copies, or modifies the Licensed Materials, or if LICENSEE transfers possession of any copy, 
adaptation, transcription or merged portion of the Licensed Materials to any other party in any way 
not expressly authorized by UMCP. 

^'^ Upon termination of this Agreement, all rights granted to LICENSEE will terminate and revert to 
UMCP. Promptly upon termination of this Agreement for any reason or upon discontinuance or 
abandonment of LICENSEE'S possession or use of the Licensed Materials, LICENSEE must 
return, or certify the destruction (and erasure from any storage device) of, all copies of the 
Licensed Matenals m LICENSEE'S possession (whether modified or unmodified), and all other 
materials pertaimng to the Licensed Materials (including all copies or portions thereof). 

5.4 Termination of this Agreement shall not relieve either party of the obligations arising hereunder 
before termination. 



SUPPORT AND OPERATTON 

6. 1 UMCP's obligation for support and maintenance of the Licensed Materials shall be limited to that 
specified herein. UMCP will support only the most current version of the Licensed Materials. 
LICENSEE is responsible for the application and implementation of new releases, computer 
program code fixes, and updates to the documentation issued to LICENSEE by UMCP. UMCP is 
not responsible for the use of superseded, outdated or uncorrected versions of the Licensed 
Matenals or for obsolescence of the Licensed Materials that may result from changes in LICEN- 
SEE s requirements or software or equipment not supplied by UMCP. 

6.2 UMCP's obligation for support and maintenance shall only apply to correction of verifiable and 
?®T?^cl"e^''^® ^ responsible for the correction of any error attributable to 
LICENSEE'S misuse or improper use of the Licensed Materials, nor shall UMCP be responsible 
for maintaining computer program code which has been modified from that delivered by UMCP 
UMCP's obligation to correct errors shall be limited to providing reasonable efforts to design and 
implement programming changes to Licensed Materials and to endeavor to correct such errors of 
which UMCP has received notice and which faU within UMCP's obligation for support as set forth 
herein. 



6.3 UMCP may from time to time during the term of this Agreement issue to LICENSEE modifica- 
aons, updates or enhancements to the Licensed Materials. All right, title and interest to any said 
materials shall be controlled by the terms and conditions of this Agreement 

PROPRIETARY PROTECTION 

7.1 LICENSEE acknowledges that UMCP is the owner of all right, title and interest in and to the 
Licensed Matenals and all modifications and enhancements thereto (including ownership of all 
trade secrets, copyrights and patents pertaining thereto and subsisting therein) subject only to the 
nghts and privileges expressly granted by UMCP. This Agreement grants LICENSEE only a right 
of limited use, revocable in accordance with the terms hereof. LICENSEE shall keep the Licensed 
Materials free and clear of all claims, liens, and encumbrances. 

7.2 LICENSEE acknowledges that the Licensed Materials are a commerciaUy valuable, proprietary 
product of UMCP, the design and development of which reflect the effort of skilled development 
experts and the mvestment of considerable time and money. UMCP claims and reserves all rights 



and benefits afforded under Federal coovrieht law anH aii !n».r«o,-^ i 

works in such portions of the LicenscdXtSs Sl^^^ ^^^^^^ ""P^^^^^ed 

a™'^^^^^^^^^^^ SslSSVi^^^^^^^^^^ ? "^^^y ^« ^-"-d Materials or 

this Ag'riement LICENSITSnit, w f^^^^^^^^ '"'^P^ ^ '^^"^"^r ^"^^^zed by 

or otherwise disseminate for anVturo^se anv nlrt nf • ^"«"5?"sept' disclose, provide 

person, firm or organiition; or (ii) eZlnle^ 1?h ^^'"""^^ other 

consistent with LICENsTe's right^ u2^^ '''''' ^«^to, 

decompile or "unlock" (as the term L eenSv '''^^ ^"^'"Pt t° 

Licensed Materials. LICENSESaU elfsS^^^^ ^ ^S®-!.^^ computer code of the 

Materials protect the Licensed M^eSL^^^^^^ ^^^"^ ^« licensed 

disclosure ^^^ensea Materials against improper use, access, duplication, dissemination or 

7.4 LICENSEE'S obligations under this Section 7 shall survive termination of this Agreement 
DISCLAIMFR OFWARR ANTTV ^^ i p I.TMITATT nMncn.^pp >^x^, 

c^J^S^c^f^^^^^^^^ °^ ^---i Materials do not infringe any 

WITH RESPFPT TO THE I Tr^cMVc'r% w ^iri;^'^^^' EXPRESS AND IMPLIED 

REPRESENTATION OR DESCRiraON^Ti^ CONFORMITY TO ANY 

miJKitn^^t^l^^^^^^^^^^^^ be Hable for any business expense, 

or any claims or demands brrughug^rLfS °' consequential damages, 

has been advised of the possibUity ofSSs ofim.nH?^^^' ^"'^^^'"e^s, even if UMCP 
claims is intended to apply withom reStoS n^^^^^^^^ a'™''''°" upon damages and 

breached or have proven ineffective provisions of this Agreement which have been 



8.1 
8.2 



9. GENHRAf. 
9.1 



9.2 
9.3 



It is agreed that the laws of the State of Maryland will Eovem ^■■■thn,^t ™f 
pnnciples. Any and all legal actions must^ hrnnoh fn ^ reference to conflict of law 
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Shall remain in fuU force and cff^t ^^^^^"^ ^ remaining temis of this Agreement 

AGREEMENT, UNDERSTANDS .T, 



9.4 



9.5 



9.6 



Agreed to; 

PATRICIA S.THOMAS 



By: 



Tide:. 



Date: ~"v*.L. -1 

— r- ■ — 



Address: 



La 



Phone: 



Accepted and .•^ritroved: 

UNIVERSp' CFN MARYLAND AT COLLEGE PARK 
By: M'l^'-'^:-^ ^ f^^-'Xr.^ 

Title; Direacr. Tecnnnology Liaison 

Date: Julh - /^^J 

Address: QScce of Technology Liaison 
-i3:2: Knox Road 
U:ave.Trsit>- of Maryland 
C:aa=2e Park, MD 20742 
3(1-^5^209 
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BEST COP! MWBLE 



SCHEDULE A 



LICENSEE Data/1 .oratintT 
LICENSEE: 



LICENSEE SITE: 



Firm: 


Address: - - /-.'. 


/-^ ■■■K . 


Contact: V-d - ' 


Phone: /. ■ ?i . 


Firm: Kw „ •- '•: 




Address: i ; ^ ,V- 


,■■0 V '■"1.1 


Contact: i ' ■ 


' > Phone: ■ > - vv< 



2. Designated Computers: 



Model # S erial # 



Operating Svs*ffrn 
Software 



3. Address for Notices; 
UMCP: 



LICENSEE: 



Office of Technology Liaison 
4312 Knox Road 
University of Maryland 
CoUege Park, MD 20742 



Designated .^^jtp; 

Address/L^catjpn nf 
Computer Tn«^fa||f^fir.f| 
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APPENDIX D 
BREAKDOWN OF CODED EVENTS BY SEARCHER 



Reference Materials Used 
Quick reference card 



Searcher Event 

1 what to do when system is searching a long time 

1 how to narrow search 

2 help with Boolean operators 
2,3,3 explanation of abbreviations 

3 information on ERIC, citations or actual articles 
6,9 just looked at it to see what it was 

7 how to limit field (publication year) 
9 how to truncate 

User manual and Reference guide (binder) 

Searcher Event 

1 how to narrow search 

1 help with Boolean 

2 help on returning to search 
2,2,3,9 just looked at it to see what it was 

Thesaurus 

Searcher Event 

2 just looked at it to see what it was 

4,6,8,9 checked for descriptors before searching 
5,5,5,7,7,8,9,9,10 checked for descriptors during search 

8 checked before searching for age-level descriptor 

Help screens 

Searcher Event-Topic selected, purpose 

1 Finding records, how to narrow search 

1 How to start search 

1»2 Guide, how to return to search 

1.7 Showing records, how to narrow search 

1»7 *Printing records, how to print selected records 

1 Show subset, how to print (mouse slipped) 

1 *Printing records, how to print 

1 Restarting session, stuck in show subset 

1 *Showing records, stuck in show subset 

2 Narrowing search, how to return to search 
2,6 Restarting session, how to return to search 
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3 Getting help, how : io print (option w/o 2nd menu) 

4,4 Printing records, faiails to select 2nd menu 

8 Fields, looking for index 

10 *Narrowing search, how to limit fields by year 

♦The searcher found information hcrce that allowed the completion of 
the task in question (five events out of total 20). 

? icon (context-sensitive help) 
Searcher ■ Event prompting sesiection of icon 



1 

4,9 
4,10 
6 
8 

10 



How to narrow sexarch 
How to nctum to ssearch 
How to print selcccxed records 
What to do when search completed 
*How to print selescted records 
How to limit by yerear 



♦Icon correctly moved to area in qcesstion and answer provided to 
complete the task (1 out of 8 total e^vcnts). 

Introductory screen 
Searcher Event 

1,2,3,8,10 Yes, read the intrcoductory screen 

4,5,6,7,9 No, did not read Ltbc introductory screen 

Macintosh instructions given b^y author 



Searcher 
1,7 
1 
1 
1 
1 
2 
4 

Index, 
Searcher 
1,2 



Event 
Use of menubar 
Close window box 

Moving mouse fromn end of mousepad 

How to delete chanracter 

Use of button to bnieak search 

How to get back innto MacSPIRS after system bomb 
How to get to hclpD secondary menus 

online 

Event 

Found when cxplcairing to narrow search, used like 
search dialog box, not helpful 



Exploration of software featurers 

Searcher Event 

1,2,2,3 ♦V/hile trying to reetum to search 

2,2,3 General exploratioon of features 

1 ♦While trying to pmint 
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♦Exploration did not provide answers to questions. 



Interface issues 

System plusses 



Searcher Event 

1,1,4,5,6,10 Interchangeability of ENTER and START SEARCH 

2,9 Boolean operators: typed 

5,8 Boolean operators: clicked 

8 Addition of previous number search when new 

search begun with Boolean operator 

2,8 # of previous search typed vs. clicked 

1 0 Ability to edit search as opposed to online 

10 % completed message when searching 

1 0 Similarity to Windows 

1 0 Boolean operators on screen a sign that software 

uses them 

1 0 Availability of search history 



System minuses 
Searcher Event 

1 Beep, no message. Show Subset selected by 

mistake and trying to show another record. 

1 Beep, no message. Tried to click Selected records 

when not available as an option. 

3 Printer error with number, but no explanation, no 

indication of what manual might contain it. 

4 Beep, no message. Tried entering multiple record 

numbers in "start printing from record #x" area in 
print dialog box. 

5 Beep, no message. Hit previous record at record 

#1 to return to search^ 

7 Message: retrieving records, please wait. No 

indication of why or % complete. Very slow. 

8 System searches for single word posting even 

though term listed in thesaurus as "young 
children" (no hyphen). 

8 Appears to scan disc for every query, discourages 

thorough searching, experimentation. 

10,10,10 Range error message, even though range used 

successfully earlier in search. Once message 
issued when search showed 99% complete. 

1 0 Search request too complex. Five terms used, but 

those with multiple words not hyphenated. 
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Intuitive navigation 



Searcher Event 

1,6 Toggle of ? icon to get cursor back 

1,1,7,9,10 Scrolling page up and down 

1,1 Next Record button/Previous Record button 

1 Toggle of radio buttons 

1 Toggle of selecting records 

2 Index, use of scroll right to view entire term 
1 0 Close window box to return to search 

Navigational errors 
Searcher Attempts to return to search from viewing records 

1 By mistake clicks in search window 

2 Clicks Previous Record back to #1, thinks index is 

search window, through help finds "Find Records". 

3,3 Tries Previous Record, then by mistake clicks in 

search window. 

4,4,4 Checks under buttons, then restarts session. 

5 Checks under buttons, tries previous record until 

beeps, finally closes window. 
7 Clear search. 

6,9,9 Clicked ?, looked under menubar, restarts session. 

9 Quits application, says it's an accident. 

Mouse problems 
Searcher Type of event 

1 (14 events) General lack of hand-eye coordination 
2,3,8 Trouble getting to secondary menus in help 

7 Initial hesitancy in use 

Problems 

Errors 

Searcher Event 

1,1 Asking to view records before search completed 

1 Typing question mark on keyboard for help 

1 Typing search word in record 

1 Clicking on Boolean operator for help in narrowing 

a search 

1 Typing search phrase in index entry 

1,3 Problem deleting with backspace 
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1 Failure too erase selected index terms before 

selecting new term 
1 Failure toc> start search before clicking on show 

records faound 

1 Retrievingg information from incorrect secondary 

menu in help due to sloppy mouse movement 
1 Selection of show subset when trying to get 

selected rrccords to print 
1 Use oi scelect fields to narrow search 

1 Back to s^tart search screen but doesn't realize it 

1 Uninteuiaonal mouse click puts word in search box 

1 Use of SBfaow Subset instead of Start Search 

1 Clicks onn search terms instead of Show Records 

Found 

2.2 Extra moouse clicks causing skipped records and 

system boomb 

2,8 Continamng to click when clock icon showing 

3 Failure tao hit Show Records Found after search 

complctce 

3 Use of SBhow Recwds Found expected to bring up 

article, onot citation 
3,3,4,5 Printing ^ records, not selected 

3.3 Breaking print while formatting 
3 Click wiiithin search history 

3 Use of C-^ps Lock to get # sign 

5 Thought print should have included descriptors 

5 Not rccoQgnizing that citations were retrieved, 

thinks noone because 0 found until last % of search 
4,5,9,9,9,9 To selecrt records, chose start printing from record 

# and osaed LaserWriter dialog box "from page to 

page* foDT from record to record. 
7,7,7 Break sesarch r -^ing no records found before 

s earc h c c omple te 
8 Trouble remembering where Print Records was 

8 Looking for index, gives up after checking under 

help for it, 
10 Failure tao deselect ? icon 

1,10 Clicks seelected records when not highlighted 

10 Start prinnt from record # fourth from the end to 

get last four records instead of selecting records. 

don't understand ^what this means** 
Searcher Process or terminology not understood 
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1 




Break search 




1.4.4 




Click on any word or drag across any phrase 
(search from records) 




1 




How to narrow search 




1,3 




Unsure of whether or not it printed 




1 




Show subset selected, not sure why can*t move 
of record 


out 


3 




Error message on using "in" Boolean operator 




3 




ERIC has citations, not articles 




3,3 




Abbreviations not obvious 




3 




Clicking in search history puts terms in search 


box 


3,3,3 


/ 


Records formatted so far when printing 




3,3 


How to search, not simple enough 




3 




Thinks it*s printing when it*s not 




5 




Zero records formatted so far 




6,8 




Not sure whether search complete when it is 




10 




Print records dialog box 




10 




Date range too wide 




10 




Complex search request (should be defined, max # 






of terms or at least more specific, i.e. use of 








hyphenated terms) 




10 




Selected records: retrieved records window 





Logic errors 



Searcher Event 

1 Selecting help topic not applicable to problem 

3 Disinclined to narrow search, pages through 228 

records 

3 Incorrect use of ''in" Boolean operator 

3 Doesn't understand difference between child 

obesity (i citation retrieved) and childhood 

obesity (2 1 retrieved) 
3 Assumes all relevant records included under one 

topic 

5,5,5,5,5 Combining search with 0 hits with another search 

7,7 Use of term twice in a search 

7 Publication year > date of database 

9 Problem with Boolean, parentheses needed to 

correctly interpret 

Response time 
Searcher Event 

1 Searcher nervous as time lengthens 
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2,3,3,3 Impatient, extra mouse clicks 

2 When printing, assumes user error when "no 

records formatted so far" appears 
7,7,8,8,10,10 Frustration with long search 
7 Frustration with length of time to retrieve record 
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